This is an automated email from the ASF dual-hosted git repository. ethanfeng pushed a commit to branch CELEBORN-166 in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
commit f54d5e5e460218e4b4238962813746adf32b143d Author: Ethan Feng <[email protected]> AuthorDate: Thu Dec 22 15:21:16 2022 +0800 [CELEBORN-166] Automate release build process. --- .gitignore | 2 + LICENSE-binary | 289 ++++++++++++++++++++++++++++++++++++++++++++++ build/append_notice.py | 63 ++++++++++ build/build-release.sh | 233 +++++++++++++++++++++++++++++++++++++ build/collect-licenses.sh | 76 ++++++++++++ build/dependency.sh | 94 +++++++++++++++ build/util.sh | 37 ++++++ dev/dependencyList | 114 ++++++++++++++++++ 8 files changed, 908 insertions(+) diff --git a/.gitignore b/.gitignore index 05fda3a0..7ab42aba 100644 --- a/.gitignore +++ b/.gitignore @@ -77,3 +77,5 @@ spark-warehouse/ logs pids +/tmp/htrace-core4-4.1.0-incubating/META-INF/NOTICE +/tmp/ diff --git a/LICENSE-binary b/LICENSE-binary new file mode 100644 index 00000000..8d6ce20f --- /dev/null +++ b/LICENSE-binary @@ -0,0 +1,289 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +------------------------------------------------------------------------------------ +This product bundles various third-party components under other open source licenses. +This section summarizes those components and their licenses. See licenses/ +for text of these licenses. + +Apache License Version 2.0 +-------------------------- +org.roaringbitmap:RoaringBitmap +org.apache.commons:commons-crypto +commons-io:commons-io +org.apache.commons:commons-lang3 +commons-logging:commons-logging +com.google.guava:guava +org.apache.hadoop:hadoop-client-api +org.apache.hadoop:hadoop-client-runtime +org.apache.htrace:htrace-core4 +org.slf4j:jcl-over-slf4j +com.google.code.findbugs:jsr305 +org.apache.logging.log4j:log4j-1.2-api +org.apache.logging.log4j:log4j-api +org.apache.logging.log4j:log4j-core +org.apache.logging.log4j:log4j-slf4j-impl +io.dropwizard.metrics:metrics-core +io.dropwizard.metrics:metrics-graphite +io.dropwizard.metrics:metrics-jvm +io.netty:netty-all +io.netty:netty-buffer +io.netty:netty-codec +io.netty:netty-codec-dns +io.netty:netty-codec-haproxy +io.netty:netty-codec-http +io.netty:netty-codec-http2 +io.netty:netty-codec-memcache +io.netty:netty-codec-mqtt +io.netty:netty-codec-redis +io.netty:netty-codec-smtp +io.netty:netty-codec-socks +io.netty:netty-codec-stomp +io.netty:netty-codec-xml +io.netty:netty-common +io.netty:netty-handler +io.netty:netty-handler-proxy +io.netty:netty-resolver +io.netty:netty-resolver-dns +io.netty:netty-resolver-dns-classes-macos +io.netty:netty-resolver-dns-native-macos +io.netty:netty-transport +io.netty:netty-transport-classes-epoll +io.netty:netty-transport-classes-kqueue +io.netty:netty-transport-native-epoll +io.netty:netty-transport-native-kqueue +io.netty:netty-transport-native-unix-common +io.netty:netty-transport-rxtx +io.netty:netty-transport-sctp +io.netty:netty-transport-udt +org.apache.ratis:ratis-client +org.apache.ratis:ratis-common +org.apache.ratis:ratis-grpc +org.apache.ratis:ratis-metrics +org.apache.ratis:ratis-netty +org.apache.ratis:ratis-proto +org.apache.ratis:ratis-server +org.apache.ratis:ratis-server-api +org.apache.ratis:ratis-thirdparty-misc +org.scala-lang:scala-library +org.scala-lang:scala-reflect +org.roaringbitmap:shims +org.yaml:snakeyaml + +COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.0 +------------ +javax.servlet:javax.servlet-api + +MIT License +------------ +org.slf4j:jul-to-slf4j +org.slf4j:slf4j-api + +COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.0 +------------ +javax.activation:activation +javax.servlet:javax.servlet-api + +BSD 3-clause +------------ +org.fusesource.leveldbjni:leveldbjni-all +com.google.protobuf:protobuf-java \ No newline at end of file diff --git a/build/append_notice.py b/build/append_notice.py new file mode 100755 index 00000000..f540ebd6 --- /dev/null +++ b/build/append_notice.py @@ -0,0 +1,63 @@ +#!/usr/bin/env python3 + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +import sys + + +def usage(): + print('Usage: %s <NOTICE file> <APPEND file1> <APPEND file2> ...' % sys.argv[0]) + + +redundant_text_list = [ + """This product includes software developed at + The Apache Software Foundation (http://www.apache.org/).""", + """This product includes software developed at + The Apache Software Foundation (https://www.apache.org/).""", + """This product includes software developed by + The Apache Software Foundation (http://www.apache.org/).""" +] + + +def append(notice_file, append_file): + with open(notice_file, 'r') as f: + notice_text = f.read() + + with open(append_file, 'r') as f: + append_text = f.read() + + for text in redundant_text_list: + append_text = append_text.replace(text, '') + + append_text = append_text.strip() + + if not append_text in notice_text: + with open(notice_file, 'a') as f: + f.write(append_text) + f.write('\n\n') + + +if __name__ == "__main__": + if len(sys.argv) < 3: + print(sys.argv) + usage() + sys.exit(-1) + + notice_file = sys.argv[1] + for append_file in sys.argv[2:]: + append(notice_file, append_file) diff --git a/build/build-release.sh b/build/build-release.sh new file mode 100755 index 00000000..1d1e2419 --- /dev/null +++ b/build/build-release.sh @@ -0,0 +1,233 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -o pipefail +set -e +set -x + +SKIP_GPG=${SKIP_GPG:-false} + +exit_with_usage() { + local NAME=$(basename $0) + cat <<EOF +Usage: $NAME <source|binary> + +Top level targets are: + source: Create source release tarball + binary: Create binary release tarball + +All other inputs are environment variables: + +RELEASE_VERSION - Release version, must match pom.xml (e.g. 1.3.0-incubating) +SKIP_GPG - (optional) Default false +EOF + exit 1 +} + +PROJECT_DIR="$( + cd "$(dirname "$0")/.." + pwd +)" +DIST_DIR="$PROJECT_DIR/dist" +NAME="incubating-bin" + +MVN="$PROJECT_DIR/build/mvn" + +SHASUM="sha512sum" +if [ "$(uname)" == "Darwin" ]; then + SHASUM="shasum -a 512" +fi + +package_binary() { + # Figure out where the RSS framework is installed + + if [ -z "$JAVA_HOME" ]; then + # Fall back on JAVA_HOME from rpm, if found + if [ $(command -v rpm) ]; then + RPM_JAVA_HOME="$(rpm -E %java_home 2>/dev/null)" + if [ "$RPM_JAVA_HOME" != "%java_home" ]; then + JAVA_HOME="$RPM_JAVA_HOME" + echo "No JAVA_HOME set, proceeding with '$JAVA_HOME' learned from rpm" + fi + fi + + if [ -z "$JAVA_HOME" ]; then + if [ $(command -v java) ]; then + # If java is in /usr/bin/java, we want /usr + JAVA_HOME="$(dirname $(dirname $(which java)))" + fi + fi + fi + + if [ -z "$JAVA_HOME" ]; then + echo "Error: JAVA_HOME is not set, cannot proceed." + exit -1 + fi + + if [ ! "$(command -v "$MVN")" ]; then + echo -e "Could not locate Maven command: '$MVN'." + exit -1 + fi + + if [ $(command -v git) ]; then + GITREV=$(git rev-parse --short HEAD 2>/dev/null || :) + if [ ! -z "$GITREV" ]; then + GITREVSTRING=" (git revision $GITREV)" + fi + unset GITREV + fi + + VERSION=$("$MVN" help:evaluate -Dexpression=project.version -Pspark-3.3 2>/dev/null | grep -v "INFO" | + grep -v "WARNING" | + tail -n 1) + SCALA_VERSION=$("$MVN" help:evaluate -Dexpression=scala.binary.version -Pspark-3.3 2>/dev/null | grep -v "INFO" | + grep -v "WARNING" | + tail -n 1) + SPARK_VERSION=$("$MVN" help:evaluate -Dexpression=spark.version -Pspark-3.3 2>/dev/null | grep -v "INFO" | + grep -v "WARNING" | + tail -n 1) + + SPARK_MAJOR_VERSION=${SPARK_VERSION%%.*} + + echo "Celeborn version is $VERSION" + + echo "Making apache-celeborn-$VERSION-$NAME.tgz" + + # Build uber fat JAR + cd "$PROJECT_DIR" + + export MAVEN_OPTS="${MAVEN_OPTS:--Xmx2g -XX:ReservedCodeCacheSize=1g}" + + # Store the command as an array because $MVN variable might have spaces in it. + # Normal quoting tricks don't work. + # See: http://mywiki.wooledge.org/BashFAQ/050 + BUILD_COMMAND=("$MVN" clean package -DskipTests -Pspark-3.3) + + # Actually build the jar + echo -e "\nBuilding with..." + echo -e "\$ ${BUILD_COMMAND[@]}\n" + + "${BUILD_COMMAND[@]}" + + # Make directories + rm -rf "$DIST_DIR" + mkdir -p "$DIST_DIR/jars" + mkdir -p "$DIST_DIR/master-jars" + mkdir -p "$DIST_DIR/worker-jars" + mkdir -p "$DIST_DIR/spark2" + mkdir -p "$DIST_DIR/spark3" + + echo "Celeborn $VERSION$GITREVSTRING" >"$DIST_DIR/RELEASE" + echo "Build flags: $@" >>"$DIST_DIR/RELEASE" + + # Copy jars + ## Copy master jars + cp "$PROJECT_DIR"/master/target/celeborn-master_$SCALA_VERSION-$VERSION.jar "$DIST_DIR/master-jars/" + cp "$PROJECT_DIR"/master/target/scala-$SCALA_VERSION/jars/*.jar "$DIST_DIR/jars/" + for jar in $(ls "$PROJECT_DIR/master/target/scala-$SCALA_VERSION/jars"); do + ( + cd $DIST_DIR/master-jars + ln -snf "../jars/$jar" . + ) + done + ## Copy worker jars + cp "$PROJECT_DIR"/worker/target/celeborn-worker_$SCALA_VERSION-$VERSION.jar "$DIST_DIR/worker-jars/" + cp "$PROJECT_DIR"/worker/target/scala-$SCALA_VERSION/jars/*.jar "$DIST_DIR/jars/" + for jar in $(ls "$PROJECT_DIR/worker/target/scala-$SCALA_VERSION/jars"); do + ( + cd $DIST_DIR/worker-jars + ln -snf "../jars/$jar" . + ) + done + ## Copy spark client jars + cp "$PROJECT_DIR"/client-spark/spark-$SPARK_MAJOR_VERSION-shaded/target/celeborn-client-spark-${SPARK_MAJOR_VERSION}-shaded_$SCALA_VERSION-$VERSION.jar "$DIST_DIR/spark3/" + #build 2.4 + + VERSION=$("$MVN" help:evaluate -Dexpression=project.version -Pspark-2.4 2>/dev/null | grep -v "INFO" | + grep -v "WARNING" | + tail -n 1) + SCALA_VERSION=$("$MVN" help:evaluate -Dexpression=scala.binary.version -Pspark-2.4 2>/dev/null | grep -v "INFO" | + grep -v "WARNING" | + tail -n 1) + SPARK_VERSION=$("$MVN" help:evaluate -Dexpression=spark.version -Pspark-2.4 2>/dev/null | grep -v "INFO" | + grep -v "WARNING" | + tail -n 1) + SPARK_MAJOR_VERSION=${SPARK_VERSION%%.*} + "$MVN" clean package -DskipTests -Pspark-2.4 + cp "$PROJECT_DIR"/client-spark/spark-$SPARK_MAJOR_VERSION-shaded/target/celeborn-client-spark-${SPARK_MAJOR_VERSION}-shaded_$SCALA_VERSION-$VERSION.jar "$DIST_DIR/spark2/" + + # Copy other things + mkdir "$DIST_DIR/conf" + cp "$PROJECT_DIR"/conf/*.template "$DIST_DIR/conf" + cp -r "$PROJECT_DIR/bin" "$DIST_DIR" + cp -r "$PROJECT_DIR/sbin" "$DIST_DIR" + mkdir "$DIST_DIR/docker" + cp "$PROJECT_DIR/docker/Dockerfile" "$DIST_DIR/docker" + cp -r "$PROJECT_DIR/docker/helm" "$DIST_DIR/docker" + # Copy notice ,license and disclaimer + cp "$PROJECT_DIR/DISCLAIMER" "$DIST_DIR/" + cp "$PROJECT_DIR/LICENSE-binary" "$DIST_DIR/LICENSE" + cp "$PROJECT_DIR/NOTICE" "$DIST_DIR/NOTICE" + ${PROJECT_DIR}/build/collect-licenses.sh $DIST_DIR/jars $DIST_DIR/ + + TARDIR_NAME="apache-celeborn-$VERSION-$NAME" + TARDIR="$PROJECT_DIR/$TARDIR_NAME" + rm -rf "$TARDIR" + cp -R "$DIST_DIR" "$TARDIR" + tar czf "apache-celeborn-$VERSION-$NAME.tgz" -C "$PROJECT_DIR" "$TARDIR_NAME" + rm -rf "$TARDIR" + + if [ "$SKIP_GPG" == "false" ]; then + gpg --armor --detach-sig "apache-celeborn-$VERSION-$NAME.tgz" + fi + $SHASUM "apache-celeborn-$VERSION-$NAME.tgz" >"apache-celeborn-$VERSION-$NAME.tgz.sha512" + +} + +package_source() { + + VERSION=$("$MVN" help:evaluate -Dexpression=project.version -Pspark-3.3 2>/dev/null | grep -v "INFO" | + grep -v "WARNING" | + tail -n 1) + + SRC_TGZ_FILE="apache-celeborn-${VERSION}-incubating-source.tgz" + SRC_TGZ="${SRC_TGZ_FILE}" + + rm -f "${SRC_TGZ}*" + + echo "Creating source release tarball ${SRC_TGZ_FILE}" + + git archive --prefix="apache-celeborn-${VERSION}-incubating-source/" -o "${SRC_TGZ}" HEAD + + if [ "$SKIP_GPG" == "false" ]; then + gpg --armor --detach-sig "${SRC_TGZ}" + fi + $SHASUM "${SRC_TGZ_FILE}" >"${SRC_TGZ_FILE}.sha512" +} + +if [[ "$1" == "source" ]]; then + package_source + exit 0 +fi + +if [[ "$1" == "binary" ]]; then + package_binary + exit 0 +fi + +exit_with_usage diff --git a/build/collect-licenses.sh b/build/collect-licenses.sh new file mode 100755 index 00000000..399625c1 --- /dev/null +++ b/build/collect-licenses.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# This script extracts from all jars in the specified directory the NOTICE files and the +# licenses folders. It then concatenates all NOTICE files and collects the contents of all +# licenses folders in the specified output directory. +# +# This tool can be used to generate a rough skeleton for the binary NOTICE file. Be aware, +# that it does not deduplicate contents. + +set -o pipefail +set -e + +SRC=${1:-.} +DST=${2:-licenses-output} +PROJECT_DIR="$( + cd "$(dirname "$0")"/.. + pwd +)" +TMP="${DST}/tmp" +NOTICE_BINARY_PREAMBLE="${PROJECT_DIR}/NOTICE" + +USAGE="$0 <SOURCE_DIRECTORY:-.> <OUTPUT_DIRECTORY:-licenses-output>" + +source "$PROJECT_DIR/build/util.sh" + +if [ "${SRC}" = "-h" ]; then + echo "${USAGE}" + exit 0 +fi + +for jar_file in $(find -L "${SRC}" -name "*.jar"); do + if [[ "$(basename ${jar_file})" != "celeborn-"* ]]; then + DIR="${TMP}/$(basename -- "${jar_file}" .jar)" + mkdir -p "${DIR}" + JAR=$(realpath "${jar_file}") + (cd "${DIR}" && jar xf ${JAR} META-INF/NOTICE META-INF/licenses) + fi +done + +NOTICE="${DST}/NOTICE" +[ -f "${NOTICE}" ] && rm "${NOTICE}" +cp "${NOTICE_BINARY_PREAMBLE}" "${NOTICE}" +( + export LC_ALL=C + find "${TMP}" -name "NOTICE*" | sort | xargs ${PROJECT_DIR}/build/append_notice.py "${NOTICE}" +) + +LICENSES="${DST}/licenses" +[ -f "${LICENSES}" ] && rm -r "${LICENSES}" +find "${TMP}" -name "licenses" -type d -exec cp -r -- "{}" "${DST}" \; + +# Search and collect license files that not bundled in any jars. +if [ ! -d ${LICENSES} ]; then + mkdir -p ${LICENSES} +fi +find "${SRC}" -name "LICENSE.*" -type f \ + ! -path "${DST}/licenses/*" ! -path "${TMP}/licenses/*" -exec cp -- "{}" "${DST}/licenses" \; + +rm -r "${TMP}" diff --git a/build/dependency.sh b/build/dependency.sh new file mode 100755 index 00000000..0b1906cb --- /dev/null +++ b/build/dependency.sh @@ -0,0 +1,94 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -o pipefail +set -e +set -x + +export LC_ALL=C + +PWD=$( + cd "$(dirname "$0")"/.. || exit + pwd +) + +MVN="${PWD}/build/mvn" + +DEP_PR="${PWD}/dev/dependencyList.tmp" +DEP="${PWD}/dev/dependencyList" + +function build_classpath() { + $MVN install -Pspark-3.3 -DskipTests + $MVN dependency:build-classpath -pl master,worker,common -Pspark-3.3 | + grep -v "INFO\|WARN" | + tail -1 | + tr ":" "\n" | + awk -F '/' '{ + artifact_id=$(NF-2); + version=$(NF-1); + jar_name=$NF; + classifier_start_index=length(artifact_id"-"version"-") + 1; + classifier_end_index=index(jar_name, ".jar") - 1; + classifier=substr(jar_name, classifier_start_index, classifier_end_index - classifier_start_index + 1); + print artifact_id"/"version"/"classifier"/"jar_name + }' | grep -v "celeborn" | sort >>"${DEP_PR}" +} + +function check_diff() { + set +e + the_diff=$(diff "${DEP}" "${DEP_PR}") + set -e + rm -rf "${DEP_PR}" + if [[ -n "${the_diff}" ]]; then + echo "Dependency List Changed Detected: " + echo "${the_diff}" + echo "To update the dependency file, run './build/dependency.sh --replace'." + exit 1 + fi +} + +rm -rf "${DEP_PR}" +cat >"${DEP_PR}" <<EOF +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +EOF + +build_classpath + +if [[ "$1" == "--replace" ]]; then + rm -rf "${DEP}" + mv "${DEP_PR}" "${DEP}" + exit 0 +fi + +check_diff diff --git a/build/util.sh b/build/util.sh new file mode 100755 index 00000000..687803d3 --- /dev/null +++ b/build/util.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +realpath() { + ( + TARGET_FILE="$1" + + cd "$(dirname "$TARGET_FILE")" + TARGET_FILE="$(basename "$TARGET_FILE")" + + COUNT=0 + while [ -L "$TARGET_FILE" -a $COUNT -lt 100 ]; do + TARGET_FILE="$(readlink "$TARGET_FILE")" + cd $(dirname "$TARGET_FILE") + TARGET_FILE="$(basename $TARGET_FILE)" + COUNT=$(($COUNT + 1)) + done + + echo "$(pwd -P)/"$TARGET_FILE"" + ) +} diff --git a/dev/dependencyList b/dev/dependencyList new file mode 100644 index 00000000..cf2e8f54 --- /dev/null +++ b/dev/dependencyList @@ -0,0 +1,114 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +RoaringBitmap/0.9.32//RoaringBitmap-0.9.32.jar +byte-buddy-agent/1.10.15//byte-buddy-agent-1.10.15.jar +byte-buddy/1.10.15//byte-buddy-1.10.15.jar +commons-crypto/1.0.0//commons-crypto-1.0.0.jar +commons-io/2.8.0//commons-io-2.8.0.jar +commons-lang3/3.10//commons-lang3-3.10.jar +commons-logging/1.1.3//commons-logging-1.1.3.jar +generics-resolver/3.0.3//generics-resolver-3.0.3.jar +guava/14.0.1//guava-14.0.1.jar +hadoop-client-api/3.2.1//hadoop-client-api-3.2.1.jar +hadoop-client-runtime/3.2.1//hadoop-client-runtime-3.2.1.jar +hamcrest-core/1.3//hamcrest-core-1.3.jar +htrace-core4/4.1.0-incubating//htrace-core4-4.1.0-incubating.jar +javax.servlet-api/3.1.0//javax.servlet-api-3.1.0.jar +jcl-over-slf4j/1.7.36//jcl-over-slf4j-1.7.36.jar +jsr305/1.3.9//jsr305-1.3.9.jar +jul-to-slf4j/1.7.36//jul-to-slf4j-1.7.36.jar +junit/4.12//junit-4.12.jar +leveldbjni-all/1.8//leveldbjni-all-1.8.jar +log4j-1.2-api/2.17.2//log4j-1.2-api-2.17.2.jar +log4j-api/2.17.2//log4j-api-2.17.2.jar +log4j-core/2.17.2//log4j-core-2.17.2.jar +log4j-slf4j-impl/2.17.2//log4j-slf4j-impl-2.17.2.jar +lz4-java/1.8.0//lz4-java-1.8.0.jar +metrics-core/3.2.6//metrics-core-3.2.6.jar +metrics-graphite/3.2.6//metrics-graphite-3.2.6.jar +metrics-jvm/3.2.6//metrics-jvm-3.2.6.jar +mockito-core/3.6.0//mockito-core-3.6.0.jar +mockito-scala-scalatest_2.12/1.16.37//mockito-scala-scalatest_2.12-1.16.37.jar +mockito-scala_2.12/1.16.37//mockito-scala_2.12-1.16.37.jar +netty-all/4.1.77.Final//netty-all-4.1.77.Final.jar +netty-buffer/4.1.77.Final//netty-buffer-4.1.77.Final.jar +netty-codec-dns/4.1.77.Final//netty-codec-dns-4.1.77.Final.jar +netty-codec-haproxy/4.1.77.Final//netty-codec-haproxy-4.1.77.Final.jar +netty-codec-http/4.1.77.Final//netty-codec-http-4.1.77.Final.jar +netty-codec-http2/4.1.77.Final//netty-codec-http2-4.1.77.Final.jar +netty-codec-memcache/4.1.77.Final//netty-codec-memcache-4.1.77.Final.jar +netty-codec-mqtt/4.1.77.Final//netty-codec-mqtt-4.1.77.Final.jar +netty-codec-redis/4.1.77.Final//netty-codec-redis-4.1.77.Final.jar +netty-codec-smtp/4.1.77.Final//netty-codec-smtp-4.1.77.Final.jar +netty-codec-socks/4.1.77.Final//netty-codec-socks-4.1.77.Final.jar +netty-codec-stomp/4.1.77.Final//netty-codec-stomp-4.1.77.Final.jar +netty-codec-xml/4.1.77.Final//netty-codec-xml-4.1.77.Final.jar +netty-codec/4.1.77.Final//netty-codec-4.1.77.Final.jar +netty-common/4.1.77.Final//netty-common-4.1.77.Final.jar +netty-handler-proxy/4.1.77.Final//netty-handler-proxy-4.1.77.Final.jar +netty-handler/4.1.77.Final//netty-handler-4.1.77.Final.jar +netty-resolver-dns-classes-macos/4.1.77.Final//netty-resolver-dns-classes-macos-4.1.77.Final.jar +netty-resolver-dns-native-macos/4.1.77.Final/osx-aarch_64/netty-resolver-dns-native-macos-4.1.77.Final-osx-aarch_64.jar +netty-resolver-dns-native-macos/4.1.77.Final/osx-x86_64/netty-resolver-dns-native-macos-4.1.77.Final-osx-x86_64.jar +netty-resolver-dns/4.1.77.Final//netty-resolver-dns-4.1.77.Final.jar +netty-resolver/4.1.77.Final//netty-resolver-4.1.77.Final.jar +netty-transport-classes-epoll/4.1.77.Final//netty-transport-classes-epoll-4.1.77.Final.jar +netty-transport-classes-kqueue/4.1.77.Final//netty-transport-classes-kqueue-4.1.77.Final.jar +netty-transport-native-epoll/4.1.77.Final/linux-aarch_64/netty-transport-native-epoll-4.1.77.Final-linux-aarch_64.jar +netty-transport-native-epoll/4.1.77.Final/linux-x86_64/netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar +netty-transport-native-kqueue/4.1.77.Final/osx-aarch_64/netty-transport-native-kqueue-4.1.77.Final-osx-aarch_64.jar +netty-transport-native-kqueue/4.1.77.Final/osx-x86_64/netty-transport-native-kqueue-4.1.77.Final-osx-x86_64.jar +netty-transport-native-unix-common/4.1.77.Final//netty-transport-native-unix-common-4.1.77.Final.jar +netty-transport-rxtx/4.1.77.Final//netty-transport-rxtx-4.1.77.Final.jar +netty-transport-sctp/4.1.77.Final//netty-transport-sctp-4.1.77.Final.jar +netty-transport-udt/4.1.77.Final//netty-transport-udt-4.1.77.Final.jar +netty-transport/4.1.77.Final//netty-transport-4.1.77.Final.jar +objenesis/3.1//objenesis-3.1.jar +protobuf-java/3.19.2//protobuf-java-3.19.2.jar +ratis-client/2.4.1//ratis-client-2.4.1.jar +ratis-common/2.4.1//ratis-common-2.4.1.jar +ratis-grpc/2.4.1//ratis-grpc-2.4.1.jar +ratis-metrics/2.4.1//ratis-metrics-2.4.1.jar +ratis-netty/2.4.1//ratis-netty-2.4.1.jar +ratis-proto/2.4.1//ratis-proto-2.4.1.jar +ratis-server-api/2.4.1//ratis-server-api-2.4.1.jar +ratis-server/2.4.1//ratis-server-2.4.1.jar +ratis-thirdparty-misc/1.0.3//ratis-thirdparty-misc-1.0.3.jar +scala-library/2.12.15//scala-library-2.12.15.jar +scala-reflect/2.12.15//scala-reflect-2.12.15.jar +scala-xml_2.12/1.2.0//scala-xml_2.12-1.2.0.jar +scalactic_2.12/3.2.7//scalactic_2.12-3.2.7.jar +scalatest-compatible/3.2.3//scalatest-compatible-3.2.3.jar +scalatest-core_2.12/3.2.3//scalatest-core_2.12-3.2.3.jar +scalatest-diagrams_2.12/3.2.3//scalatest-diagrams_2.12-3.2.3.jar +scalatest-featurespec_2.12/3.2.3//scalatest-featurespec_2.12-3.2.3.jar +scalatest-flatspec_2.12/3.2.3//scalatest-flatspec_2.12-3.2.3.jar +scalatest-freespec_2.12/3.2.3//scalatest-freespec_2.12-3.2.3.jar +scalatest-funspec_2.12/3.2.3//scalatest-funspec_2.12-3.2.3.jar +scalatest-funsuite_2.12/3.2.3//scalatest-funsuite_2.12-3.2.3.jar +scalatest-matchers-core_2.12/3.2.3//scalatest-matchers-core_2.12-3.2.3.jar +scalatest-mustmatchers_2.12/3.2.3//scalatest-mustmatchers_2.12-3.2.3.jar +scalatest-propspec_2.12/3.2.3//scalatest-propspec_2.12-3.2.3.jar +scalatest-refspec_2.12/3.2.3//scalatest-refspec_2.12-3.2.3.jar +scalatest-shouldmatchers_2.12/3.2.3//scalatest-shouldmatchers_2.12-3.2.3.jar +scalatest-wordspec_2.12/3.2.3//scalatest-wordspec_2.12-3.2.3.jar +scalatest_2.12/3.2.3//scalatest_2.12-3.2.3.jar +shims/0.9.32//shims-0.9.32.jar +slf4j-api/1.7.36//slf4j-api-1.7.36.jar +snakeyaml/1.30//snakeyaml-1.30.jar +zstd-jni/1.5.2-1//zstd-jni-1.5.2-1.jar
