[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[ https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635388#comment-16635388 ] ASF GitHub Bot commented on FLINK-8819: --- zentol closed pull request #6642: [FLINK-8819][travis] Rework travis script to use stages URL: https://github.com/apache/flink/pull/6642 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/.travis.yml b/.travis.yml index 47ccf421555..28b7e4c752e 100644 --- a/.travis.yml +++ b/.travis.yml @@ -20,8 +20,11 @@ sudo: required dist: trusty cache: + # default timeout is too low + timeout: 600 directories: - $HOME/.m2 + - $HOME/flink_cache # do not cache our own artifacts before_cache: @@ -31,62 +34,6 @@ install: true language: java -# - define unique cache names in case the auto-generated ones are not unique -# (see https://docs.travis-ci.com/user/caching/#Caches-and-build-matrices) -# - See https://issues.apache.org/jira/browse/FLINK-1072 -matrix: - include: -- jdk: "oraclejdk8" - env: -- TEST="core" -- PROFILE="-Dhadoop.version=2.8.3" -- CACHE_NAME=JDK8_H280_CO -- jdk: "oraclejdk8" - env: -- TEST="libraries" -- PROFILE="-Dhadoop.version=2.8.3" -- CACHE_NAME=JDK8_H280_L -- jdk: "oraclejdk8" - env: -- TEST="connectors" -- PROFILE="-Dhadoop.version=2.8.3 -Pinclude-kinesis" -- CACHE_NAME=JDK8_H280_CN -- jdk: "oraclejdk8" - env: -- TEST="tests" -- PROFILE="-Dhadoop.version=2.8.3" -- CACHE_NAME=JDK8_H280_T -- jdk: "oraclejdk8" - env: -- TEST="misc" -- PROFILE="-Dhadoop.version=2.8.3 -Dinclude_hadoop_aws" -- CACHE_NAME=JDK8_H280_M -- jdk: "openjdk8" - env: -- TEST="core" -- PROFILE="-Dhadoop.version=2.4.1" -- CACHE_NAME=JDK8_H241_CO -- jdk: "openjdk8" - env: -- TEST="libraries" -- PROFILE="-Dhadoop.version=2.4.1" -- CACHE_NAME=JDK8_H241_L -- jdk: "openjdk8" - env: -- TEST="connectors" -- PROFILE="-Dhadoop.version=2.4.1 -Pinclude-kinesis" -- CACHE_NAME=JDK8_H241_CN -- jdk: "openjdk8" - env: -- TEST="tests" -- PROFILE="-Dhadoop.version=2.4.1" -- CACHE_NAME=JDK8_H241_T -- jdk: "openjdk8" - env: -- TEST="misc" -- PROFILE="-Dhadoop.version=2.4.1" -- CACHE_NAME=JDK8_H241_M - git: depth: 100 @@ -102,6 +49,9 @@ env: before_script: - "gem install --no-document --version 0.8.9 faraday " + - "export -f travis_nanoseconds" + - "export -f travis_time_start" + - "export -f travis_time_finish" # Install maven 3.2.5 since trusty uses 3.3.9 for which shading is broken before_install: @@ -119,6 +69,57 @@ before_install: - chmod +x docker-compose - sudo mv docker-compose /usr/local/bin -# We run mvn and monitor its output. If there is no output for the specified number of seconds, we -# print the stack traces of all running Java processes. -script: "./tools/travis_mvn_watchdog.sh 300" + +jdk: "oraclejdk8" +jobs: + include: +# main profile +- stage: compile + script: ./tools/travis_controller.sh + env: PROFILE="-Dhadoop.version=2.8.3 -Pinclude-kinesis -Dinclude_hadoop_aws" + name: compile +- stage: test + script: ./tools/travis_controller.sh + env: PROFILE="-Dhadoop.version=2.8.3 -Pinclude-kinesis -Dinclude_hadoop_aws" + name: core +- script: ./tools/travis_controller.sh + env: PROFILE="-Dhadoop.version=2.8.3 -Pinclude-kinesis -Dinclude_hadoop_aws" + name: libraries +- script: ./tools/travis_controller.sh + env: PROFILE="-Dhadoop.version=2.8.3 -Pinclude-kinesis -Dinclude_hadoop_aws" + name: connectors +- script: ./tools/travis_controller.sh + env: PROFILE="-Dhadoop.version=2.8.3 -Pinclude-kinesis -Dinclude_hadoop_aws" + name: tests +- script: ./tools/travis_controller.sh + env: PROFILE="-Dhadoop.version=2.8.3 -Pinclude-kinesis -Dinclude_hadoop_aws" + name: misc +- stage: cleanup + script: ./tools/travis_controller.sh + env: PROFILE="-Dhadoop.version=2.8.3 -Pinclude-kinesis -Dinclude_hadoop_aws" + name: cleanup +# legacy profile +- stage: compile + script: ./tools/travis_controller.sh + env: PROFILE="-Dhadoop.version=2.4.1 -Pinclude-kinesis -DlegacyCode" + name: compile(legacy) +- stage: test + script: ./tools/travis_controller.sh + env: PROFILE="-Dhadoop.version=2.4.1 -Pinclude-kinesis -DlegacyCode" + name: core(legacy) +- script: ./tools/travis_controller.sh + env: PROFILE="-Dhadoop.version=2.4.1
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[
https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627052#comment-16627052
]
ASF GitHub Bot commented on FLINK-8819:
---
zentol commented on a change in pull request #6642: [FLINK-8819][travis] Rework
travis script to use stages
URL: https://github.com/apache/flink/pull/6642#discussion_r220119351
##
File path: tools/travis_controller.sh
##
@@ -0,0 +1,211 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+CACHE_DIR="$HOME/flink_cache"
+CACHE_BUILD_DIR="$CACHE_DIR/$TRAVIS_BUILD_NUMBER"
+CACHE_FLINK_DIR="$CACHE_BUILD_DIR/flink"
+
+HERE="`dirname \"$0\"`"# relative
+HERE="`( cd \"$HERE\" && pwd )`" # absolutized and normalized
+if [ -z "$HERE" ] ; then
+ # error; for some reason, the path is not accessible
+ # to the script (e.g. permissions re-evaled after suid)
+ exit 1 # fail
+fi
+
+source "${HERE}/travis/fold.sh"
+source "${HERE}/travis/stage.sh"
+source "${HERE}/travis/shade.sh"
+
+function deleteOldCaches() {
+ while read CACHE_DIR; do
+ local old_number="${CACHE_DIR##*/}"
+ if [ "$old_number" -lt "$TRAVIS_BUILD_NUMBER" ]; then
+ echo "Deleting old cache $CACHE_DIR"
+ rm -rf "$CACHE_DIR"
+ fi
+ done
+}
+
+# delete leftover caches from previous builds
+find "$CACHE_DIR" -mindepth 1 -maxdepth 1 | grep -v "$TRAVIS_BUILD_NUMBER" |
deleteOldCaches
+
+function getCurrentStage() {
+ STAGE_NUMBER=$(echo "$TRAVIS_JOB_NUMBER" | cut -d'.' -f 2)
+ case $STAGE_NUMBER in
+ (1)
+ echo "$STAGE_COMPILE"
+ ;;
+ (2)
+ echo "$STAGE_COMPILE"
+ ;;
+ (3)
+ echo "$STAGE_CORE"
+ ;;
+ (4)
+ echo "$STAGE_LIBRARIES"
+ ;;
+ (5)
+ echo "$STAGE_CONNECTORS"
+ ;;
+ (6)
+ echo "$STAGE_TESTS"
+ ;;
+ (7)
+ echo "$STAGE_MISC"
+ ;;
+ (8)
+ echo "$STAGE_CORE"
+ ;;
+ (9)
+ echo "$STAGE_LIBRARIES"
+ ;;
+ (10)
+ echo "$STAGE_CONNECTORS"
+ ;;
+ (11)
+ echo "$STAGE_TESTS"
+ ;;
+ (12)
+ echo "$STAGE_MISC"
+ ;;
+ (13)
+ echo "$STAGE_CLEANUP"
+ ;;
+ (14)
+ echo "$STAGE_CLEANUP"
+ ;;
+ (*)
+ echo "Invalid stage detected ($STAGE_NUMBER)"
+ return 1
+ ;;
+ esac
+
+ return 0
+}
+
+STAGE=$(getCurrentStage)
+if [ $? != 0 ]; then
+ echo "Could not determine current stage."
+ exit 1
+fi
+echo "Current stage: \"$STAGE\""
+
+EXIT_CODE=0
+
+# Run actual compile steps
+if [ $STAGE == "$STAGE_COMPILE" ]; then
+ MVN="mvn clean install -nsu -Dflink.forkCount=2
-Dflink.forkCountTestPackage=2 -Dmaven.javadoc.skip=true -B -DskipTests
$PROFILE"
+ $MVN
+ EXIT_CODE=$?
+
+if [ $EXIT_CODE == 0 ]; then
+printf
"\n\n==\n"
+printf "Checking dependency convergence\n"
+printf
"==\n"
+
+./tools/check_dependency_convergence.sh
+EXIT_CODE=$?
+else
+printf
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[
https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626305#comment-16626305
]
ASF GitHub Bot commented on FLINK-8819:
---
GJL commented on a change in pull request #6642: [FLINK-8819][travis] Rework
travis script to use stages
URL: https://github.com/apache/flink/pull/6642#discussion_r219956271
##
File path: tools/travis_controller.sh
##
@@ -0,0 +1,211 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+CACHE_DIR="$HOME/flink_cache"
+CACHE_BUILD_DIR="$CACHE_DIR/$TRAVIS_BUILD_NUMBER"
+CACHE_FLINK_DIR="$CACHE_BUILD_DIR/flink"
+
+HERE="`dirname \"$0\"`"# relative
+HERE="`( cd \"$HERE\" && pwd )`" # absolutized and normalized
+if [ -z "$HERE" ] ; then
+ # error; for some reason, the path is not accessible
+ # to the script (e.g. permissions re-evaled after suid)
+ exit 1 # fail
+fi
+
+source "${HERE}/travis/fold.sh"
+source "${HERE}/travis/stage.sh"
+source "${HERE}/travis/shade.sh"
+
+function deleteOldCaches() {
+ while read CACHE_DIR; do
+ local old_number="${CACHE_DIR##*/}"
+ if [ "$old_number" -lt "$TRAVIS_BUILD_NUMBER" ]; then
+ echo "Deleting old cache $CACHE_DIR"
+ rm -rf "$CACHE_DIR"
+ fi
+ done
+}
+
+# delete leftover caches from previous builds
+find "$CACHE_DIR" -mindepth 1 -maxdepth 1 | grep -v "$TRAVIS_BUILD_NUMBER" |
deleteOldCaches
+
+function getCurrentStage() {
+ STAGE_NUMBER=$(echo "$TRAVIS_JOB_NUMBER" | cut -d'.' -f 2)
+ case $STAGE_NUMBER in
+ (1)
+ echo "$STAGE_COMPILE"
+ ;;
+ (2)
+ echo "$STAGE_COMPILE"
+ ;;
+ (3)
+ echo "$STAGE_CORE"
+ ;;
+ (4)
+ echo "$STAGE_LIBRARIES"
+ ;;
+ (5)
+ echo "$STAGE_CONNECTORS"
+ ;;
+ (6)
+ echo "$STAGE_TESTS"
+ ;;
+ (7)
+ echo "$STAGE_MISC"
+ ;;
+ (8)
+ echo "$STAGE_CORE"
+ ;;
+ (9)
+ echo "$STAGE_LIBRARIES"
+ ;;
+ (10)
+ echo "$STAGE_CONNECTORS"
+ ;;
+ (11)
+ echo "$STAGE_TESTS"
+ ;;
+ (12)
+ echo "$STAGE_MISC"
+ ;;
+ (13)
+ echo "$STAGE_CLEANUP"
+ ;;
+ (14)
+ echo "$STAGE_CLEANUP"
+ ;;
+ (*)
+ echo "Invalid stage detected ($STAGE_NUMBER)"
+ return 1
+ ;;
+ esac
+
+ return 0
+}
+
+STAGE=$(getCurrentStage)
+if [ $? != 0 ]; then
+ echo "Could not determine current stage."
+ exit 1
+fi
+echo "Current stage: \"$STAGE\""
+
+EXIT_CODE=0
+
+# Run actual compile steps
+if [ $STAGE == "$STAGE_COMPILE" ]; then
+ MVN="mvn clean install -nsu -Dflink.forkCount=2
-Dflink.forkCountTestPackage=2 -Dmaven.javadoc.skip=true -B -DskipTests
$PROFILE"
+ $MVN
+ EXIT_CODE=$?
+
+if [ $EXIT_CODE == 0 ]; then
+printf
"\n\n==\n"
+printf "Checking dependency convergence\n"
+printf
"==\n"
+
+./tools/check_dependency_convergence.sh
+EXIT_CODE=$?
+else
+printf
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[
https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626303#comment-16626303
]
ASF GitHub Bot commented on FLINK-8819:
---
GJL commented on a change in pull request #6642: [FLINK-8819][travis] Rework
travis script to use stages
URL: https://github.com/apache/flink/pull/6642#discussion_r219956271
##
File path: tools/travis_controller.sh
##
@@ -0,0 +1,211 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+CACHE_DIR="$HOME/flink_cache"
+CACHE_BUILD_DIR="$CACHE_DIR/$TRAVIS_BUILD_NUMBER"
+CACHE_FLINK_DIR="$CACHE_BUILD_DIR/flink"
+
+HERE="`dirname \"$0\"`"# relative
+HERE="`( cd \"$HERE\" && pwd )`" # absolutized and normalized
+if [ -z "$HERE" ] ; then
+ # error; for some reason, the path is not accessible
+ # to the script (e.g. permissions re-evaled after suid)
+ exit 1 # fail
+fi
+
+source "${HERE}/travis/fold.sh"
+source "${HERE}/travis/stage.sh"
+source "${HERE}/travis/shade.sh"
+
+function deleteOldCaches() {
+ while read CACHE_DIR; do
+ local old_number="${CACHE_DIR##*/}"
+ if [ "$old_number" -lt "$TRAVIS_BUILD_NUMBER" ]; then
+ echo "Deleting old cache $CACHE_DIR"
+ rm -rf "$CACHE_DIR"
+ fi
+ done
+}
+
+# delete leftover caches from previous builds
+find "$CACHE_DIR" -mindepth 1 -maxdepth 1 | grep -v "$TRAVIS_BUILD_NUMBER" |
deleteOldCaches
+
+function getCurrentStage() {
+ STAGE_NUMBER=$(echo "$TRAVIS_JOB_NUMBER" | cut -d'.' -f 2)
+ case $STAGE_NUMBER in
+ (1)
+ echo "$STAGE_COMPILE"
+ ;;
+ (2)
+ echo "$STAGE_COMPILE"
+ ;;
+ (3)
+ echo "$STAGE_CORE"
+ ;;
+ (4)
+ echo "$STAGE_LIBRARIES"
+ ;;
+ (5)
+ echo "$STAGE_CONNECTORS"
+ ;;
+ (6)
+ echo "$STAGE_TESTS"
+ ;;
+ (7)
+ echo "$STAGE_MISC"
+ ;;
+ (8)
+ echo "$STAGE_CORE"
+ ;;
+ (9)
+ echo "$STAGE_LIBRARIES"
+ ;;
+ (10)
+ echo "$STAGE_CONNECTORS"
+ ;;
+ (11)
+ echo "$STAGE_TESTS"
+ ;;
+ (12)
+ echo "$STAGE_MISC"
+ ;;
+ (13)
+ echo "$STAGE_CLEANUP"
+ ;;
+ (14)
+ echo "$STAGE_CLEANUP"
+ ;;
+ (*)
+ echo "Invalid stage detected ($STAGE_NUMBER)"
+ return 1
+ ;;
+ esac
+
+ return 0
+}
+
+STAGE=$(getCurrentStage)
+if [ $? != 0 ]; then
+ echo "Could not determine current stage."
+ exit 1
+fi
+echo "Current stage: \"$STAGE\""
+
+EXIT_CODE=0
+
+# Run actual compile steps
+if [ $STAGE == "$STAGE_COMPILE" ]; then
+ MVN="mvn clean install -nsu -Dflink.forkCount=2
-Dflink.forkCountTestPackage=2 -Dmaven.javadoc.skip=true -B -DskipTests
$PROFILE"
+ $MVN
+ EXIT_CODE=$?
+
+if [ $EXIT_CODE == 0 ]; then
+printf
"\n\n==\n"
+printf "Checking dependency convergence\n"
+printf
"==\n"
+
+./tools/check_dependency_convergence.sh
+EXIT_CODE=$?
+else
+printf
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[
https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626299#comment-16626299
]
ASF GitHub Bot commented on FLINK-8819:
---
GJL commented on a change in pull request #6642: [FLINK-8819][travis] Rework
travis script to use stages
URL: https://github.com/apache/flink/pull/6642#discussion_r219955526
##
File path: tools/travis_controller.sh
##
@@ -0,0 +1,211 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+CACHE_DIR="$HOME/flink_cache"
+CACHE_BUILD_DIR="$CACHE_DIR/$TRAVIS_BUILD_NUMBER"
+CACHE_FLINK_DIR="$CACHE_BUILD_DIR/flink"
+
+HERE="`dirname \"$0\"`"# relative
+HERE="`( cd \"$HERE\" && pwd )`" # absolutized and normalized
+if [ -z "$HERE" ] ; then
+ # error; for some reason, the path is not accessible
+ # to the script (e.g. permissions re-evaled after suid)
+ exit 1 # fail
+fi
+
+source "${HERE}/travis/fold.sh"
+source "${HERE}/travis/stage.sh"
+source "${HERE}/travis/shade.sh"
+
+function deleteOldCaches() {
+ while read CACHE_DIR; do
+ local old_number="${CACHE_DIR##*/}"
+ if [ "$old_number" -lt "$TRAVIS_BUILD_NUMBER" ]; then
+ echo "Deleting old cache $CACHE_DIR"
+ rm -rf "$CACHE_DIR"
+ fi
+ done
+}
+
+# delete leftover caches from previous builds
+find "$CACHE_DIR" -mindepth 1 -maxdepth 1 | grep -v "$TRAVIS_BUILD_NUMBER" |
deleteOldCaches
+
+function getCurrentStage() {
+ STAGE_NUMBER=$(echo "$TRAVIS_JOB_NUMBER" | cut -d'.' -f 2)
+ case $STAGE_NUMBER in
+ (1)
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Rework travis script to use build stages
>
>
> Key: FLINK-8819
> URL: https://issues.apache.org/jira/browse/FLINK-8819
> Project: Flink
> Issue Type: Sub-task
> Components: Build System, Travis
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
> Labels: pull-request-available
>
> This issue is for tracking efforts to rework our Travis scripts to use
> [stages|https://docs.travis-ci.com/user/build-stages/].
> This feature allows us to define a sequence of jobs that are run one after
> another. This implies that we can define dependencies between jobs, in
> contrast to our existing jobs that have to be self-contained.
> As an example, we could have a compile stage, and a test stage with multiple
> jobs.
> The main benefit here is that we no longer have to compile modules multiple
> times, which would reduce our build times.
> The major issue here however is that there is no _proper_ support for passing
> build-artifacts from one stage to the next. According to this
> [issue|https://github.com/travis-ci/beta-features/issues/28] it is on their
> to-do-list however.
> In the mean-time we could manually transfer the artifacts between stages by
> either using the Travis cache or some other external storage. The cache
> solution would work by setting up a cached directory (just like the mvn
> cache) and creating build-scope directories within containing the artifacts
> (I have a prototype that works like this).
> The major concern here is that of cleaning up the cache/storage.
> We can clean things up if
> * our script fails
> * the last stage succeeds.
> We can *not* clean things up if
> * the build is canceled
> * travis fails the build due to a timeout or similar
> as apparently there is [no way to run a script at the end of a
>
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[
https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626300#comment-16626300
]
ASF GitHub Bot commented on FLINK-8819:
---
GJL commented on a change in pull request #6642: [FLINK-8819][travis] Rework
travis script to use stages
URL: https://github.com/apache/flink/pull/6642#discussion_r219955579
##
File path: tools/travis_mvn_watchdog.sh
##
@@ -160,10 +55,11 @@ esac
# -nsu option forbids downloading snapshot artifacts. The only snapshot
artifacts we depend are from
# Flink, which however should all be built locally. see FLINK-7230
MVN_LOGGING_OPTIONS="-Dlog.dir=${ARTIFACTS_DIR}
-Dlog4j.configuration=file://$LOG4J_PROPERTIES
-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn"
-MVN_COMMON_OPTIONS="-nsu -Dflink.forkCount=2 -Dflink.forkCountTestPackage=2 -B
$MVN_LOGGING_OPTIONS"
-MVN_COMPILE_OPTIONS="$MVN_COMPILE_OPTIONS -DskipTests"
+MVN_COMMON_OPTIONS="-nsu -Dflink.forkCount=2 -Dflink.forkCountTestPackage=2
-Dfast -B $MVN_LOGGING_OPTIONS"
+MVN_COMPILE_OPTIONS="-DskipTests"
+MVN_TEST_OPTIONS="$MVN_LOGGING_OPTIONS"
-MVN_COMPILE="mvn $MVN_COMMON_OPTIONS $MVN_COMPILE_OPTIONS $PROFILE
$MVN_COMPILE_MODULES clean install"
+MVN_COMPILE="mvn $MVN_COMMON_OPTIONS $MVN_COMPILE_OPTIONS $PROFILE
$MVN_COMPILE_MODULES install"
MVN_TEST="mvn $MVN_COMMON_OPTIONS $MVN_TEST_OPTIONS $PROFILE $MVN_TEST_MODULES
verify"
MVN_PID="${ARTIFACTS_DIR}/watchdog.mvn.pid"
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Rework travis script to use build stages
>
>
> Key: FLINK-8819
> URL: https://issues.apache.org/jira/browse/FLINK-8819
> Project: Flink
> Issue Type: Sub-task
> Components: Build System, Travis
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
> Labels: pull-request-available
>
> This issue is for tracking efforts to rework our Travis scripts to use
> [stages|https://docs.travis-ci.com/user/build-stages/].
> This feature allows us to define a sequence of jobs that are run one after
> another. This implies that we can define dependencies between jobs, in
> contrast to our existing jobs that have to be self-contained.
> As an example, we could have a compile stage, and a test stage with multiple
> jobs.
> The main benefit here is that we no longer have to compile modules multiple
> times, which would reduce our build times.
> The major issue here however is that there is no _proper_ support for passing
> build-artifacts from one stage to the next. According to this
> [issue|https://github.com/travis-ci/beta-features/issues/28] it is on their
> to-do-list however.
> In the mean-time we could manually transfer the artifacts between stages by
> either using the Travis cache or some other external storage. The cache
> solution would work by setting up a cached directory (just like the mvn
> cache) and creating build-scope directories within containing the artifacts
> (I have a prototype that works like this).
> The major concern here is that of cleaning up the cache/storage.
> We can clean things up if
> * our script fails
> * the last stage succeeds.
> We can *not* clean things up if
> * the build is canceled
> * travis fails the build due to a timeout or similar
> as apparently there is [no way to run a script at the end of a
> build|https://github.com/travis-ci/travis-ci/issues/4221].
> Thus we would either have to periodically clear the cache, or encode more
> information into the cached files that would allow _other_ builds to clean up
> stale date. (For example the build number or date).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[
https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625780#comment-16625780
]
ASF GitHub Bot commented on FLINK-8819:
---
zentol commented on a change in pull request #6642: [FLINK-8819][travis] Rework
travis script to use stages
URL: https://github.com/apache/flink/pull/6642#discussion_r219824599
##
File path: tools/travis_controller.sh
##
@@ -0,0 +1,211 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+CACHE_DIR="$HOME/flink_cache"
+CACHE_BUILD_DIR="$CACHE_DIR/$TRAVIS_BUILD_NUMBER"
+CACHE_FLINK_DIR="$CACHE_BUILD_DIR/flink"
+
+HERE="`dirname \"$0\"`"# relative
+HERE="`( cd \"$HERE\" && pwd )`" # absolutized and normalized
+if [ -z "$HERE" ] ; then
+ # error; for some reason, the path is not accessible
+ # to the script (e.g. permissions re-evaled after suid)
+ exit 1 # fail
+fi
+
+source "${HERE}/travis/fold.sh"
+source "${HERE}/travis/stage.sh"
+source "${HERE}/travis/shade.sh"
+
+function deleteOldCaches() {
+ while read CACHE_DIR; do
+ local old_number="${CACHE_DIR##*/}"
+ if [ "$old_number" -lt "$TRAVIS_BUILD_NUMBER" ]; then
+ echo "Deleting old cache $CACHE_DIR"
+ rm -rf "$CACHE_DIR"
+ fi
+ done
+}
+
+# delete leftover caches from previous builds
+find "$CACHE_DIR" -mindepth 1 -maxdepth 1 | grep -v "$TRAVIS_BUILD_NUMBER" |
deleteOldCaches
+
+function getCurrentStage() {
+ STAGE_NUMBER=$(echo "$TRAVIS_JOB_NUMBER" | cut -d'.' -f 2)
+ case $STAGE_NUMBER in
+ (1)
+ echo "$STAGE_COMPILE"
+ ;;
+ (2)
+ echo "$STAGE_COMPILE"
+ ;;
+ (3)
+ echo "$STAGE_CORE"
+ ;;
+ (4)
+ echo "$STAGE_LIBRARIES"
+ ;;
+ (5)
+ echo "$STAGE_CONNECTORS"
+ ;;
+ (6)
+ echo "$STAGE_TESTS"
+ ;;
+ (7)
+ echo "$STAGE_MISC"
+ ;;
+ (8)
+ echo "$STAGE_CORE"
+ ;;
+ (9)
+ echo "$STAGE_LIBRARIES"
+ ;;
+ (10)
+ echo "$STAGE_CONNECTORS"
+ ;;
+ (11)
+ echo "$STAGE_TESTS"
+ ;;
+ (12)
+ echo "$STAGE_MISC"
+ ;;
+ (13)
+ echo "$STAGE_CLEANUP"
+ ;;
+ (14)
+ echo "$STAGE_CLEANUP"
+ ;;
+ (*)
+ echo "Invalid stage detected ($STAGE_NUMBER)"
+ return 1
+ ;;
+ esac
+
+ return 0
+}
+
+STAGE=$(getCurrentStage)
+if [ $? != 0 ]; then
+ echo "Could not determine current stage."
+ exit 1
+fi
+echo "Current stage: \"$STAGE\""
+
+EXIT_CODE=0
+
+# Run actual compile steps
+if [ $STAGE == "$STAGE_COMPILE" ]; then
+ MVN="mvn clean install -nsu -Dflink.forkCount=2
-Dflink.forkCountTestPackage=2 -Dmaven.javadoc.skip=true -B -DskipTests
$PROFILE"
+ $MVN
+ EXIT_CODE=$?
+
+if [ $EXIT_CODE == 0 ]; then
+printf
"\n\n==\n"
+printf "Checking dependency convergence\n"
+printf
"==\n"
+
+./tools/check_dependency_convergence.sh
+EXIT_CODE=$?
+else
+printf
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[
https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625778#comment-16625778
]
ASF GitHub Bot commented on FLINK-8819:
---
zentol commented on a change in pull request #6642: [FLINK-8819][travis] Rework
travis script to use stages
URL: https://github.com/apache/flink/pull/6642#discussion_r219801639
##
File path: tools/travis_mvn_watchdog.sh
##
@@ -160,10 +55,11 @@ esac
# -nsu option forbids downloading snapshot artifacts. The only snapshot
artifacts we depend are from
# Flink, which however should all be built locally. see FLINK-7230
MVN_LOGGING_OPTIONS="-Dlog.dir=${ARTIFACTS_DIR}
-Dlog4j.configuration=file://$LOG4J_PROPERTIES
-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn"
-MVN_COMMON_OPTIONS="-nsu -Dflink.forkCount=2 -Dflink.forkCountTestPackage=2 -B
$MVN_LOGGING_OPTIONS"
-MVN_COMPILE_OPTIONS="$MVN_COMPILE_OPTIONS -DskipTests"
+MVN_COMMON_OPTIONS="-nsu -Dflink.forkCount=2 -Dflink.forkCountTestPackage=2
-Dfast -B $MVN_LOGGING_OPTIONS"
+MVN_COMPILE_OPTIONS="-DskipTests"
+MVN_TEST_OPTIONS="$MVN_LOGGING_OPTIONS"
-MVN_COMPILE="mvn $MVN_COMMON_OPTIONS $MVN_COMPILE_OPTIONS $PROFILE
$MVN_COMPILE_MODULES clean install"
+MVN_COMPILE="mvn $MVN_COMMON_OPTIONS $MVN_COMPILE_OPTIONS $PROFILE
$MVN_COMPILE_MODULES install"
MVN_TEST="mvn $MVN_COMMON_OPTIONS $MVN_TEST_OPTIONS $PROFILE $MVN_TEST_MODULES
verify"
MVN_PID="${ARTIFACTS_DIR}/watchdog.mvn.pid"
Review comment:
true, I think there are also other places where we could, but I'd prefer to
do that in a follow-up. So far i only touched code in `travis_mvn_watchdog`
that i had to. Moving the code into separate files was just to prevent creating
another gigantic uber-script, and not meant as a general cleanup of this file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Rework travis script to use build stages
>
>
> Key: FLINK-8819
> URL: https://issues.apache.org/jira/browse/FLINK-8819
> Project: Flink
> Issue Type: Sub-task
> Components: Build System, Travis
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
> Labels: pull-request-available
>
> This issue is for tracking efforts to rework our Travis scripts to use
> [stages|https://docs.travis-ci.com/user/build-stages/].
> This feature allows us to define a sequence of jobs that are run one after
> another. This implies that we can define dependencies between jobs, in
> contrast to our existing jobs that have to be self-contained.
> As an example, we could have a compile stage, and a test stage with multiple
> jobs.
> The main benefit here is that we no longer have to compile modules multiple
> times, which would reduce our build times.
> The major issue here however is that there is no _proper_ support for passing
> build-artifacts from one stage to the next. According to this
> [issue|https://github.com/travis-ci/beta-features/issues/28] it is on their
> to-do-list however.
> In the mean-time we could manually transfer the artifacts between stages by
> either using the Travis cache or some other external storage. The cache
> solution would work by setting up a cached directory (just like the mvn
> cache) and creating build-scope directories within containing the artifacts
> (I have a prototype that works like this).
> The major concern here is that of cleaning up the cache/storage.
> We can clean things up if
> * our script fails
> * the last stage succeeds.
> We can *not* clean things up if
> * the build is canceled
> * travis fails the build due to a timeout or similar
> as apparently there is [no way to run a script at the end of a
> build|https://github.com/travis-ci/travis-ci/issues/4221].
> Thus we would either have to periodically clear the cache, or encode more
> information into the cached files that would allow _other_ builds to clean up
> stale date. (For example the build number or date).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[
https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625666#comment-16625666
]
ASF GitHub Bot commented on FLINK-8819:
---
zentol commented on a change in pull request #6642: [FLINK-8819][travis] Rework
travis script to use stages
URL: https://github.com/apache/flink/pull/6642#discussion_r219801639
##
File path: tools/travis_mvn_watchdog.sh
##
@@ -160,10 +55,11 @@ esac
# -nsu option forbids downloading snapshot artifacts. The only snapshot
artifacts we depend are from
# Flink, which however should all be built locally. see FLINK-7230
MVN_LOGGING_OPTIONS="-Dlog.dir=${ARTIFACTS_DIR}
-Dlog4j.configuration=file://$LOG4J_PROPERTIES
-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn"
-MVN_COMMON_OPTIONS="-nsu -Dflink.forkCount=2 -Dflink.forkCountTestPackage=2 -B
$MVN_LOGGING_OPTIONS"
-MVN_COMPILE_OPTIONS="$MVN_COMPILE_OPTIONS -DskipTests"
+MVN_COMMON_OPTIONS="-nsu -Dflink.forkCount=2 -Dflink.forkCountTestPackage=2
-Dfast -B $MVN_LOGGING_OPTIONS"
+MVN_COMPILE_OPTIONS="-DskipTests"
+MVN_TEST_OPTIONS="$MVN_LOGGING_OPTIONS"
-MVN_COMPILE="mvn $MVN_COMMON_OPTIONS $MVN_COMPILE_OPTIONS $PROFILE
$MVN_COMPILE_MODULES clean install"
+MVN_COMPILE="mvn $MVN_COMMON_OPTIONS $MVN_COMPILE_OPTIONS $PROFILE
$MVN_COMPILE_MODULES install"
MVN_TEST="mvn $MVN_COMMON_OPTIONS $MVN_TEST_OPTIONS $PROFILE $MVN_TEST_MODULES
verify"
MVN_PID="${ARTIFACTS_DIR}/watchdog.mvn.pid"
Review comment:
true, I think there are also other places where we could, but I'd prefer to
do that in a follow-up. So far i only touched code in `travis_mvn_watchdog`
that i had to. Moving the code into separate files was just to create another
gigantic uber-script, and not meant as a general cleanup of this file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Rework travis script to use build stages
>
>
> Key: FLINK-8819
> URL: https://issues.apache.org/jira/browse/FLINK-8819
> Project: Flink
> Issue Type: Sub-task
> Components: Build System, Travis
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
> Labels: pull-request-available
>
> This issue is for tracking efforts to rework our Travis scripts to use
> [stages|https://docs.travis-ci.com/user/build-stages/].
> This feature allows us to define a sequence of jobs that are run one after
> another. This implies that we can define dependencies between jobs, in
> contrast to our existing jobs that have to be self-contained.
> As an example, we could have a compile stage, and a test stage with multiple
> jobs.
> The main benefit here is that we no longer have to compile modules multiple
> times, which would reduce our build times.
> The major issue here however is that there is no _proper_ support for passing
> build-artifacts from one stage to the next. According to this
> [issue|https://github.com/travis-ci/beta-features/issues/28] it is on their
> to-do-list however.
> In the mean-time we could manually transfer the artifacts between stages by
> either using the Travis cache or some other external storage. The cache
> solution would work by setting up a cached directory (just like the mvn
> cache) and creating build-scope directories within containing the artifacts
> (I have a prototype that works like this).
> The major concern here is that of cleaning up the cache/storage.
> We can clean things up if
> * our script fails
> * the last stage succeeds.
> We can *not* clean things up if
> * the build is canceled
> * travis fails the build due to a timeout or similar
> as apparently there is [no way to run a script at the end of a
> build|https://github.com/travis-ci/travis-ci/issues/4221].
> Thus we would either have to periodically clear the cache, or encode more
> information into the cached files that would allow _other_ builds to clean up
> stale date. (For example the build number or date).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[ https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625606#comment-16625606 ] ASF GitHub Bot commented on FLINK-8819: --- zentol commented on a change in pull request #6642: [FLINK-8819][travis] Rework travis script to use stages URL: https://github.com/apache/flink/pull/6642#discussion_r219787240 ## File path: tools/travis/fold.sh ## @@ -0,0 +1,33 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +FOLD_ESCAPE="\x0d\x1b" Review comment: sure This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Rework travis script to use build stages > > > Key: FLINK-8819 > URL: https://issues.apache.org/jira/browse/FLINK-8819 > Project: Flink > Issue Type: Sub-task > Components: Build System, Travis >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Trivial > Labels: pull-request-available > > This issue is for tracking efforts to rework our Travis scripts to use > [stages|https://docs.travis-ci.com/user/build-stages/]. > This feature allows us to define a sequence of jobs that are run one after > another. This implies that we can define dependencies between jobs, in > contrast to our existing jobs that have to be self-contained. > As an example, we could have a compile stage, and a test stage with multiple > jobs. > The main benefit here is that we no longer have to compile modules multiple > times, which would reduce our build times. > The major issue here however is that there is no _proper_ support for passing > build-artifacts from one stage to the next. According to this > [issue|https://github.com/travis-ci/beta-features/issues/28] it is on their > to-do-list however. > In the mean-time we could manually transfer the artifacts between stages by > either using the Travis cache or some other external storage. The cache > solution would work by setting up a cached directory (just like the mvn > cache) and creating build-scope directories within containing the artifacts > (I have a prototype that works like this). > The major concern here is that of cleaning up the cache/storage. > We can clean things up if > * our script fails > * the last stage succeeds. > We can *not* clean things up if > * the build is canceled > * travis fails the build due to a timeout or similar > as apparently there is [no way to run a script at the end of a > build|https://github.com/travis-ci/travis-ci/issues/4221]. > Thus we would either have to periodically clear the cache, or encode more > information into the cached files that would allow _other_ builds to clean up > stale date. (For example the build number or date). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[
https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625605#comment-16625605
]
ASF GitHub Bot commented on FLINK-8819:
---
zentol commented on a change in pull request #6642: [FLINK-8819][travis] Rework
travis script to use stages
URL: https://github.com/apache/flink/pull/6642#discussion_r219787204
##
File path: tools/travis_controller.sh
##
@@ -0,0 +1,211 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+CACHE_DIR="$HOME/flink_cache"
+CACHE_BUILD_DIR="$CACHE_DIR/$TRAVIS_BUILD_NUMBER"
+CACHE_FLINK_DIR="$CACHE_BUILD_DIR/flink"
+
+HERE="`dirname \"$0\"`"# relative
+HERE="`( cd \"$HERE\" && pwd )`" # absolutized and normalized
+if [ -z "$HERE" ] ; then
+ # error; for some reason, the path is not accessible
+ # to the script (e.g. permissions re-evaled after suid)
+ exit 1 # fail
+fi
+
+source "${HERE}/travis/fold.sh"
+source "${HERE}/travis/stage.sh"
+source "${HERE}/travis/shade.sh"
+
+function deleteOldCaches() {
+ while read CACHE_DIR; do
+ local old_number="${CACHE_DIR##*/}"
+ if [ "$old_number" -lt "$TRAVIS_BUILD_NUMBER" ]; then
+ echo "Deleting old cache $CACHE_DIR"
+ rm -rf "$CACHE_DIR"
+ fi
+ done
+}
+
+# delete leftover caches from previous builds
+find "$CACHE_DIR" -mindepth 1 -maxdepth 1 | grep -v "$TRAVIS_BUILD_NUMBER" |
deleteOldCaches
+
+function getCurrentStage() {
+ STAGE_NUMBER=$(echo "$TRAVIS_JOB_NUMBER" | cut -d'.' -f 2)
+ case $STAGE_NUMBER in
+ (1)
Review comment:
do you mean as an env variable in `.travis.yml`? That wouldn't work since
then the caches are no longer shared.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Rework travis script to use build stages
>
>
> Key: FLINK-8819
> URL: https://issues.apache.org/jira/browse/FLINK-8819
> Project: Flink
> Issue Type: Sub-task
> Components: Build System, Travis
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
> Labels: pull-request-available
>
> This issue is for tracking efforts to rework our Travis scripts to use
> [stages|https://docs.travis-ci.com/user/build-stages/].
> This feature allows us to define a sequence of jobs that are run one after
> another. This implies that we can define dependencies between jobs, in
> contrast to our existing jobs that have to be self-contained.
> As an example, we could have a compile stage, and a test stage with multiple
> jobs.
> The main benefit here is that we no longer have to compile modules multiple
> times, which would reduce our build times.
> The major issue here however is that there is no _proper_ support for passing
> build-artifacts from one stage to the next. According to this
> [issue|https://github.com/travis-ci/beta-features/issues/28] it is on their
> to-do-list however.
> In the mean-time we could manually transfer the artifacts between stages by
> either using the Travis cache or some other external storage. The cache
> solution would work by setting up a cached directory (just like the mvn
> cache) and creating build-scope directories within containing the artifacts
> (I have a prototype that works like this).
> The major concern here is that of cleaning up the cache/storage.
> We can clean things up if
> * our script fails
> * the last stage succeeds.
> We can *not* clean things up if
> * the build is canceled
> * travis fails the build due to a timeout or
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[
https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625520#comment-16625520
]
ASF GitHub Bot commented on FLINK-8819:
---
zentol commented on a change in pull request #6642: [FLINK-8819][travis] Rework
travis script to use stages
URL: https://github.com/apache/flink/pull/6642#discussion_r219759044
##
File path: tools/travis_controller.sh
##
@@ -0,0 +1,211 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+CACHE_DIR="$HOME/flink_cache"
+CACHE_BUILD_DIR="$CACHE_DIR/$TRAVIS_BUILD_NUMBER"
+CACHE_FLINK_DIR="$CACHE_BUILD_DIR/flink"
+
+HERE="`dirname \"$0\"`"# relative
+HERE="`( cd \"$HERE\" && pwd )`" # absolutized and normalized
+if [ -z "$HERE" ] ; then
+ # error; for some reason, the path is not accessible
+ # to the script (e.g. permissions re-evaled after suid)
+ exit 1 # fail
+fi
+
+source "${HERE}/travis/fold.sh"
+source "${HERE}/travis/stage.sh"
+source "${HERE}/travis/shade.sh"
+
+function deleteOldCaches() {
+ while read CACHE_DIR; do
+ local old_number="${CACHE_DIR##*/}"
+ if [ "$old_number" -lt "$TRAVIS_BUILD_NUMBER" ]; then
+ echo "Deleting old cache $CACHE_DIR"
+ rm -rf "$CACHE_DIR"
+ fi
+ done
+}
+
+# delete leftover caches from previous builds
+find "$CACHE_DIR" -mindepth 1 -maxdepth 1 | grep -v "$TRAVIS_BUILD_NUMBER" |
deleteOldCaches
+
+function getCurrentStage() {
+ STAGE_NUMBER=$(echo "$TRAVIS_JOB_NUMBER" | cut -d'.' -f 2)
+ case $STAGE_NUMBER in
+ (1)
+ echo "$STAGE_COMPILE"
+ ;;
+ (2)
+ echo "$STAGE_COMPILE"
+ ;;
+ (3)
+ echo "$STAGE_CORE"
+ ;;
+ (4)
+ echo "$STAGE_LIBRARIES"
+ ;;
+ (5)
+ echo "$STAGE_CONNECTORS"
+ ;;
+ (6)
+ echo "$STAGE_TESTS"
+ ;;
+ (7)
+ echo "$STAGE_MISC"
+ ;;
+ (8)
+ echo "$STAGE_CORE"
+ ;;
+ (9)
+ echo "$STAGE_LIBRARIES"
+ ;;
+ (10)
+ echo "$STAGE_CONNECTORS"
+ ;;
+ (11)
+ echo "$STAGE_TESTS"
+ ;;
+ (12)
+ echo "$STAGE_MISC"
+ ;;
+ (13)
+ echo "$STAGE_CLEANUP"
+ ;;
+ (14)
+ echo "$STAGE_CLEANUP"
+ ;;
+ (*)
+ echo "Invalid stage detected ($STAGE_NUMBER)"
+ return 1
+ ;;
+ esac
+
+ return 0
+}
+
+STAGE=$(getCurrentStage)
+if [ $? != 0 ]; then
+ echo "Could not determine current stage."
+ exit 1
+fi
+echo "Current stage: \"$STAGE\""
+
+EXIT_CODE=0
+
+# Run actual compile steps
+if [ $STAGE == "$STAGE_COMPILE" ]; then
+ MVN="mvn clean install -nsu -Dflink.forkCount=2
-Dflink.forkCountTestPackage=2 -Dmaven.javadoc.skip=true -B -DskipTests
$PROFILE"
+ $MVN
+ EXIT_CODE=$?
+
+if [ $EXIT_CODE == 0 ]; then
+printf
"\n\n==\n"
+printf "Checking dependency convergence\n"
+printf
"==\n"
+
+./tools/check_dependency_convergence.sh
+EXIT_CODE=$?
+else
+printf
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[
https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609082#comment-16609082
]
ASF GitHub Bot commented on FLINK-8819:
---
GJL commented on a change in pull request #6642: [FLINK-8819][travis] Rework
travis script to use stages
URL: https://github.com/apache/flink/pull/6642#discussion_r216296870
##
File path: tools/travis_controller.sh
##
@@ -0,0 +1,211 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+CACHE_DIR="$HOME/flink_cache"
+CACHE_BUILD_DIR="$CACHE_DIR/$TRAVIS_BUILD_NUMBER"
+CACHE_FLINK_DIR="$CACHE_BUILD_DIR/flink"
+
+HERE="`dirname \"$0\"`"# relative
+HERE="`( cd \"$HERE\" && pwd )`" # absolutized and normalized
+if [ -z "$HERE" ] ; then
+ # error; for some reason, the path is not accessible
+ # to the script (e.g. permissions re-evaled after suid)
+ exit 1 # fail
+fi
+
+source "${HERE}/travis/fold.sh"
+source "${HERE}/travis/stage.sh"
+source "${HERE}/travis/shade.sh"
+
+function deleteOldCaches() {
+ while read CACHE_DIR; do
+ local old_number="${CACHE_DIR##*/}"
+ if [ "$old_number" -lt "$TRAVIS_BUILD_NUMBER" ]; then
+ echo "Deleting old cache $CACHE_DIR"
+ rm -rf "$CACHE_DIR"
+ fi
+ done
+}
+
+# delete leftover caches from previous builds
+find "$CACHE_DIR" -mindepth 1 -maxdepth 1 | grep -v "$TRAVIS_BUILD_NUMBER" |
deleteOldCaches
+
+function getCurrentStage() {
+ STAGE_NUMBER=$(echo "$TRAVIS_JOB_NUMBER" | cut -d'.' -f 2)
+ case $STAGE_NUMBER in
+ (1)
+ echo "$STAGE_COMPILE"
+ ;;
+ (2)
+ echo "$STAGE_COMPILE"
+ ;;
+ (3)
+ echo "$STAGE_CORE"
+ ;;
+ (4)
+ echo "$STAGE_LIBRARIES"
+ ;;
+ (5)
+ echo "$STAGE_CONNECTORS"
+ ;;
+ (6)
+ echo "$STAGE_TESTS"
+ ;;
+ (7)
+ echo "$STAGE_MISC"
+ ;;
+ (8)
+ echo "$STAGE_CORE"
+ ;;
+ (9)
+ echo "$STAGE_LIBRARIES"
+ ;;
+ (10)
+ echo "$STAGE_CONNECTORS"
+ ;;
+ (11)
+ echo "$STAGE_TESTS"
+ ;;
+ (12)
+ echo "$STAGE_MISC"
+ ;;
+ (13)
+ echo "$STAGE_CLEANUP"
+ ;;
+ (14)
+ echo "$STAGE_CLEANUP"
+ ;;
+ (*)
+ echo "Invalid stage detected ($STAGE_NUMBER)"
+ return 1
+ ;;
+ esac
+
+ return 0
+}
+
+STAGE=$(getCurrentStage)
+if [ $? != 0 ]; then
+ echo "Could not determine current stage."
+ exit 1
+fi
+echo "Current stage: \"$STAGE\""
+
+EXIT_CODE=0
+
+# Run actual compile steps
+if [ $STAGE == "$STAGE_COMPILE" ]; then
+ MVN="mvn clean install -nsu -Dflink.forkCount=2
-Dflink.forkCountTestPackage=2 -Dmaven.javadoc.skip=true -B -DskipTests
$PROFILE"
+ $MVN
+ EXIT_CODE=$?
+
+if [ $EXIT_CODE == 0 ]; then
+printf
"\n\n==\n"
+printf "Checking dependency convergence\n"
+printf
"==\n"
+
+./tools/check_dependency_convergence.sh
+EXIT_CODE=$?
+else
+printf
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[
https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608551#comment-16608551
]
ASF GitHub Bot commented on FLINK-8819:
---
GJL commented on a change in pull request #6642: [FLINK-8819][travis] Rework
travis script to use stages
URL: https://github.com/apache/flink/pull/6642#discussion_r216162537
##
File path: tools/travis_mvn_watchdog.sh
##
@@ -160,10 +55,11 @@ esac
# -nsu option forbids downloading snapshot artifacts. The only snapshot
artifacts we depend are from
# Flink, which however should all be built locally. see FLINK-7230
MVN_LOGGING_OPTIONS="-Dlog.dir=${ARTIFACTS_DIR}
-Dlog4j.configuration=file://$LOG4J_PROPERTIES
-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn"
-MVN_COMMON_OPTIONS="-nsu -Dflink.forkCount=2 -Dflink.forkCountTestPackage=2 -B
$MVN_LOGGING_OPTIONS"
-MVN_COMPILE_OPTIONS="$MVN_COMPILE_OPTIONS -DskipTests"
+MVN_COMMON_OPTIONS="-nsu -Dflink.forkCount=2 -Dflink.forkCountTestPackage=2
-Dfast -B $MVN_LOGGING_OPTIONS"
+MVN_COMPILE_OPTIONS="-DskipTests"
+MVN_TEST_OPTIONS="$MVN_LOGGING_OPTIONS"
-MVN_COMPILE="mvn $MVN_COMMON_OPTIONS $MVN_COMPILE_OPTIONS $PROFILE
$MVN_COMPILE_MODULES clean install"
+MVN_COMPILE="mvn $MVN_COMMON_OPTIONS $MVN_COMPILE_OPTIONS $PROFILE
$MVN_COMPILE_MODULES install"
MVN_TEST="mvn $MVN_COMMON_OPTIONS $MVN_TEST_OPTIONS $PROFILE $MVN_TEST_MODULES
verify"
MVN_PID="${ARTIFACTS_DIR}/watchdog.mvn.pid"
Review comment:
In `print_system_info` of this file, we could use the `start_fold` and
`end_fold` functions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Rework travis script to use build stages
>
>
> Key: FLINK-8819
> URL: https://issues.apache.org/jira/browse/FLINK-8819
> Project: Flink
> Issue Type: Sub-task
> Components: Build System, Travis
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
> Labels: pull-request-available
>
> This issue is for tracking efforts to rework our Travis scripts to use
> [stages|https://docs.travis-ci.com/user/build-stages/].
> This feature allows us to define a sequence of jobs that are run one after
> another. This implies that we can define dependencies between jobs, in
> contrast to our existing jobs that have to be self-contained.
> As an example, we could have a compile stage, and a test stage with multiple
> jobs.
> The main benefit here is that we no longer have to compile modules multiple
> times, which would reduce our build times.
> The major issue here however is that there is no _proper_ support for passing
> build-artifacts from one stage to the next. According to this
> [issue|https://github.com/travis-ci/beta-features/issues/28] it is on their
> to-do-list however.
> In the mean-time we could manually transfer the artifacts between stages by
> either using the Travis cache or some other external storage. The cache
> solution would work by setting up a cached directory (just like the mvn
> cache) and creating build-scope directories within containing the artifacts
> (I have a prototype that works like this).
> The major concern here is that of cleaning up the cache/storage.
> We can clean things up if
> * our script fails
> * the last stage succeeds.
> We can *not* clean things up if
> * the build is canceled
> * travis fails the build due to a timeout or similar
> as apparently there is [no way to run a script at the end of a
> build|https://github.com/travis-ci/travis-ci/issues/4221].
> Thus we would either have to periodically clear the cache, or encode more
> information into the cached files that would allow _other_ builds to clean up
> stale date. (For example the build number or date).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[
https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608549#comment-16608549
]
ASF GitHub Bot commented on FLINK-8819:
---
GJL commented on a change in pull request #6642: [FLINK-8819][travis] Rework
travis script to use stages
URL: https://github.com/apache/flink/pull/6642#discussion_r216167556
##
File path: tools/travis_controller.sh
##
@@ -0,0 +1,211 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+CACHE_DIR="$HOME/flink_cache"
+CACHE_BUILD_DIR="$CACHE_DIR/$TRAVIS_BUILD_NUMBER"
+CACHE_FLINK_DIR="$CACHE_BUILD_DIR/flink"
+
+HERE="`dirname \"$0\"`"# relative
+HERE="`( cd \"$HERE\" && pwd )`" # absolutized and normalized
+if [ -z "$HERE" ] ; then
+ # error; for some reason, the path is not accessible
+ # to the script (e.g. permissions re-evaled after suid)
+ exit 1 # fail
+fi
+
+source "${HERE}/travis/fold.sh"
+source "${HERE}/travis/stage.sh"
+source "${HERE}/travis/shade.sh"
+
+function deleteOldCaches() {
+ while read CACHE_DIR; do
+ local old_number="${CACHE_DIR##*/}"
+ if [ "$old_number" -lt "$TRAVIS_BUILD_NUMBER" ]; then
+ echo "Deleting old cache $CACHE_DIR"
+ rm -rf "$CACHE_DIR"
+ fi
+ done
+}
+
+# delete leftover caches from previous builds
+find "$CACHE_DIR" -mindepth 1 -maxdepth 1 | grep -v "$TRAVIS_BUILD_NUMBER" |
deleteOldCaches
+
+function getCurrentStage() {
+ STAGE_NUMBER=$(echo "$TRAVIS_JOB_NUMBER" | cut -d'.' -f 2)
+ case $STAGE_NUMBER in
+ (1)
Review comment:
It seems to be fragile to rely on the stage number (reordering stages in
`.travis.yml`, adding stages in the middle). Why not set the stage constants
(`$STAGE_COMPILE`, `$STAGE_CORE`, etc.) as an env variable, e.g.,
`CURRENT_STAGE`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Rework travis script to use build stages
>
>
> Key: FLINK-8819
> URL: https://issues.apache.org/jira/browse/FLINK-8819
> Project: Flink
> Issue Type: Sub-task
> Components: Build System, Travis
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
> Labels: pull-request-available
>
> This issue is for tracking efforts to rework our Travis scripts to use
> [stages|https://docs.travis-ci.com/user/build-stages/].
> This feature allows us to define a sequence of jobs that are run one after
> another. This implies that we can define dependencies between jobs, in
> contrast to our existing jobs that have to be self-contained.
> As an example, we could have a compile stage, and a test stage with multiple
> jobs.
> The main benefit here is that we no longer have to compile modules multiple
> times, which would reduce our build times.
> The major issue here however is that there is no _proper_ support for passing
> build-artifacts from one stage to the next. According to this
> [issue|https://github.com/travis-ci/beta-features/issues/28] it is on their
> to-do-list however.
> In the mean-time we could manually transfer the artifacts between stages by
> either using the Travis cache or some other external storage. The cache
> solution would work by setting up a cached directory (just like the mvn
> cache) and creating build-scope directories within containing the artifacts
> (I have a prototype that works like this).
> The major concern here is that of cleaning up the cache/storage.
> We can clean things up if
> * our script fails
> * the last stage
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[ https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608550#comment-16608550 ] ASF GitHub Bot commented on FLINK-8819: --- GJL commented on a change in pull request #6642: [FLINK-8819][travis] Rework travis script to use stages URL: https://github.com/apache/flink/pull/6642#discussion_r216166677 ## File path: tools/travis/fold.sh ## @@ -0,0 +1,33 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +FOLD_ESCAPE="\x0d\x1b" Review comment: It wasn't done before but can we document the origins of these constants? It seems to be an undocumented travis feature. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Rework travis script to use build stages > > > Key: FLINK-8819 > URL: https://issues.apache.org/jira/browse/FLINK-8819 > Project: Flink > Issue Type: Sub-task > Components: Build System, Travis >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Trivial > Labels: pull-request-available > > This issue is for tracking efforts to rework our Travis scripts to use > [stages|https://docs.travis-ci.com/user/build-stages/]. > This feature allows us to define a sequence of jobs that are run one after > another. This implies that we can define dependencies between jobs, in > contrast to our existing jobs that have to be self-contained. > As an example, we could have a compile stage, and a test stage with multiple > jobs. > The main benefit here is that we no longer have to compile modules multiple > times, which would reduce our build times. > The major issue here however is that there is no _proper_ support for passing > build-artifacts from one stage to the next. According to this > [issue|https://github.com/travis-ci/beta-features/issues/28] it is on their > to-do-list however. > In the mean-time we could manually transfer the artifacts between stages by > either using the Travis cache or some other external storage. The cache > solution would work by setting up a cached directory (just like the mvn > cache) and creating build-scope directories within containing the artifacts > (I have a prototype that works like this). > The major concern here is that of cleaning up the cache/storage. > We can clean things up if > * our script fails > * the last stage succeeds. > We can *not* clean things up if > * the build is canceled > * travis fails the build due to a timeout or similar > as apparently there is [no way to run a script at the end of a > build|https://github.com/travis-ci/travis-ci/issues/4221]. > Thus we would either have to periodically clear the cache, or encode more > information into the cached files that would allow _other_ builds to clean up > stale date. (For example the build number or date). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[ https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598680#comment-16598680 ] ASF GitHub Bot commented on FLINK-8819: --- zentol opened a new pull request #6642: [FLINK-8819][travis] Rework travis script to use stages URL: https://github.com/apache/flink/pull/6642 ## What is the purpose of the change This PR reworks the travis scripts to use stages. Stages allow jobs to be organized in sequential steps, in contrast to the current approach of all jobs running in parallel. This allows jobs to depend on each other, with the obvious use-case of separating code compilation and test execution. A subsequent stage is only executed if the previous stage has completed successfully, in that all builds in the stage have completed successfully. In other words, if checkstyle fails, no tests are executed, so be mindful of that. The benefit here really is that we no longer compile (parts of) Flink in each profile, and move part of the compilation overhead into a separate profile. We don't decrease the total runtime due to added overhead (upload/download of cache), but the individual builds are faster, and more manageable in the long-term. An example build can be seen here: https://travis-ci.org/zentol/flink/builds/422925766 ## High-level overview The new scripts define 3 stages: Compile, Test and Cleanup. In the compile stage we compile Flink and run QA checks like checkstyle. The compiled Flink project is placed into the travis cache to make it accessible to subsequent builds. The test stage consists of 5 jobs based on our existing test splitting (core, libs, connectors, tests, misc). These builds retrieve the compiled Flink version from the cache, install it into the local repository and subsequently run the tests. The cleanup jobs deletes the compiled Flink artifact from the cache. This step isn't exactly necessary, but still nice to have. Some additional small refactorings have been made to separate `travis_mvn_watchdog.sh` into individual parts, which we can build on in the future. ## Low-level details ### Caching The downside of stages is there is no easy-to-use way to pass on build artifacts. The caching approach _works_ but has the caveat that builds have to share the same cache. The travis cache is only shared between builds if the build configurations are identical; most notably they can't call different scripts nor have different environment variables. As a workaround we map the `TRAVIS_JOB_NUMBER` to a specific stage. (If you look at the build linked in the PR, `4583.1` would be the value I'm talking about). The order of jobs is deterministic, so for example we always know that `1-2` belong to the compile stage, with `2` always being configured for the legacy codebase. ### travis_controller All stage-related logic is handled by the `travis_controller` script. In short: * it determines where we are in the build process based on `TRAVIS_JOB_NUMBER` * if in compile step * remove existing cached flink versions (fail-safe cleanup to prevent cache from growing larger over time) * compile Flink and do QA checks (shading, dependency convergence, checkstyle etc.) * copy flink to cache location * drop unnecessary files (like original jars) from compiled version * if in test step * fetch flink from cache * update all timestamps to prevent compiler plugins from recompiling classes * execute `travis_mvn_watchdog.sh` * if in cleanup step * well, cleanup stuff ### travis_mvn_watchdog Despite the above changes `travis_mvn_watchdog.sh` works pretty much like it did before. It first `install`s Flink (except now without `clean` as this would remove already compiled classes) and then runs `mvn verify`. This has the downside that we still package jars twice, which actually takes a while. We could skip this in theory by directly invoking the `surefire` plugin, but various issue in our build/tests prevent this from working at the moment. And I don't want to delay this change further. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Rework travis script to use build stages > > > Key: FLINK-8819 > URL: https://issues.apache.org/jira/browse/FLINK-8819 > Project: Flink > Issue Type: Sub-task > Components: Build System, Travis >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority:
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[ https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387528#comment-16387528 ] Chesnay Schepler commented on FLINK-8819: - Yes. The "compile" stage also covers the various checks we have (rat, checkstyle, spotbugs, ...). The test stages disable all of them...except dependency-convergence (forgot about that one). > Rework travis script to use build stages > > > Key: FLINK-8819 > URL: https://issues.apache.org/jira/browse/FLINK-8819 > Project: Flink > Issue Type: Sub-task > Components: Build System, Travis >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Trivial > > This issue is for tracking efforts to rework our Travis scripts to use > [stages|https://docs.travis-ci.com/user/build-stages/]. > This feature allows us to define a sequence of jobs that are run one after > another. This implies that we can define dependencies between jobs, in > contrast to our existing jobs that have to be self-contained. > As an example, we could have a compile stage, and a test stage with multiple > jobs. > The main benefit here is that we no longer have to compile modules multiple > times, which would reduce our build times. > The major issue here however is that there is no _proper_ support for passing > build-artifacts from one stage to the next. According to this > [issue|https://github.com/travis-ci/beta-features/issues/28] it is on their > to-do-list however. > In the mean-time we could manually transfer the artifacts between stages by > either using the Travis cache or some other external storage. The cache > solution would work by setting up a cached directory (just like the mvn > cache) and creating build-scope directories within containing the artifacts > (I have a prototype that works like this). > The major concern here is that of cleaning up the cache/storage. > We can clean things up if > * our script fails > * the last stage succeeds. > We can *not* clean things up if > * the build is canceled > * travis fails the build due to a timeout or similar > as apparently there is [no way to run a script at the end of a > build|https://github.com/travis-ci/travis-ci/issues/4221]. > Thus we would either have to periodically clear the cache, or encode more > information into the cached files that would allow _other_ builds to clean up > stale date. (For example the build number or date). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[ https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387522#comment-16387522 ] Stephan Ewen commented on FLINK-8819: - I agree, already helps. Does the "compile" stage also cover checkstyle? > Rework travis script to use build stages > > > Key: FLINK-8819 > URL: https://issues.apache.org/jira/browse/FLINK-8819 > Project: Flink > Issue Type: Sub-task > Components: Build System, Travis >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Trivial > > This issue is for tracking efforts to rework our Travis scripts to use > [stages|https://docs.travis-ci.com/user/build-stages/]. > This feature allows us to define a sequence of jobs that are run one after > another. This implies that we can define dependencies between jobs, in > contrast to our existing jobs that have to be self-contained. > As an example, we could have a compile stage, and a test stage with multiple > jobs. > The main benefit here is that we no longer have to compile modules multiple > times, which would reduce our build times. > The major issue here however is that there is no _proper_ support for passing > build-artifacts from one stage to the next. According to this > [issue|https://github.com/travis-ci/beta-features/issues/28] it is on their > to-do-list however. > In the mean-time we could manually transfer the artifacts between stages by > either using the Travis cache or some other external storage. The cache > solution would work by setting up a cached directory (just like the mvn > cache) and creating build-scope directories within containing the artifacts > (I have a prototype that works like this). > The major concern here is that of cleaning up the cache/storage. > We can clean things up if > * our script fails > * the last stage succeeds. > We can *not* clean things up if > * the build is canceled > * travis fails the build due to a timeout or similar > as apparently there is [no way to run a script at the end of a > build|https://github.com/travis-ci/travis-ci/issues/4221]. > Thus we would either have to periodically clear the cache, or encode more > information into the cached files that would allow _other_ builds to clean up > stale date. (For example the build number or date). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[ https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384864#comment-16384864 ] Chesnay Schepler commented on FLINK-8819: - A full green build can be found [here|https://travis-ci.org/zentol/flink/builds/348724165]. The total build times hasn't changed (+-5%); all the time we save by avoiding the redundant compiles is eaten up by the additional overhead of stages, like the downloading/unpacking of the cache). The job times have generally improved however, the "misc" profile, which currently starts scratching the 50m limit, now only takes 27m. This may be enough justification to do the switch, even though we didn't accomplish the desired goal. > Rework travis script to use build stages > > > Key: FLINK-8819 > URL: https://issues.apache.org/jira/browse/FLINK-8819 > Project: Flink > Issue Type: Sub-task > Components: Build System, Travis >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Trivial > > This issue is for tracking efforts to rework our Travis scripts to use > [stages|https://docs.travis-ci.com/user/build-stages/]. > This feature allows us to define a sequence of jobs that are run one after > another. This implies that we can define dependencies between jobs, in > contrast to our existing jobs that have to be self-contained. > As an example, we could have a compile stage, and a test stage with multiple > jobs. > The main benefit here is that we no longer have to compile modules multiple > times, which would reduce our build times. > The major issue here however is that there is no _proper_ support for passing > build-artifacts from one stage to the next. According to this > [issue|https://github.com/travis-ci/beta-features/issues/28] it is on their > to-do-list however. > In the mean-time we could manually transfer the artifacts between stages by > either using the Travis cache or some other external storage. The cache > solution would work by setting up a cached directory (just like the mvn > cache) and creating build-scope directories within containing the artifacts > (I have a prototype that works like this). > The major concern here is that of cleaning up the cache/storage. > We can clean things up if > * our script fails > * the last stage succeeds. > We can *not* clean things up if > * the build is canceled > * travis fails the build due to a timeout or similar > as apparently there is [no way to run a script at the end of a > build|https://github.com/travis-ci/travis-ci/issues/4221]. > Thus we would either have to periodically clear the cache, or encode more > information into the cached files that would allow _other_ builds to clean up > stale date. (For example the build number or date). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8819) Rework travis script to use build stages
[ https://issues.apache.org/jira/browse/FLINK-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382609#comment-16382609 ] Chesnay Schepler commented on FLINK-8819: - A nearly finished prototype can be found [here|https://github.com/zentol/flink/tree/travis_stage_prototype], and an example build [here|https://travis-ci.org/zentol/flink/builds/347908225]. > Rework travis script to use build stages > > > Key: FLINK-8819 > URL: https://issues.apache.org/jira/browse/FLINK-8819 > Project: Flink > Issue Type: Sub-task > Components: Build System, Travis >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Trivial > > This issue is for tracking efforts to rework our Travis scripts to use > [stages|https://docs.travis-ci.com/user/build-stages/]. > This feature allows us to define a sequence of jobs that are run one after > another. This implies that we can define dependencies between jobs, in > contrast to our existing jobs that have to be self-contained. > As an example, we could have a compile stage, and a test stage with multiple > jobs. > The main benefit here is that we no longer have to compile modules multiple > times, which would reduce our build times. > The major issue here however is that there is no _proper_ support for passing > build-artifacts from one stage to the next. According to this > [issue|https://github.com/travis-ci/beta-features/issues/28] it is on their > to-do-list however. > In the mean-time we could manually transfer the artifacts between stages by > either using the Travis cache or some other external storage. The cache > solution would work by setting up a cached directory (just like the mvn > cache) and creating build-scope directories within containing the artifacts > (I have a prototype that works like this). > The major concern here is that of cleaning up the cache/storage. > We can clean things up if > * our script fails > * the last stage succeeds. > We can *not* clean things up if > * the build is canceled > * travis fails the build due to a timeout or similar > as apparently there is [no way to run a script at the end of a > build|https://github.com/travis-ci/travis-ci/issues/4221]. > Thus we would either have to periodically clear the cache, or encode more > information into the cached files that would allow _other_ builds to clean up > stale date. (For example the build number or date). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
