[ 
https://issues.apache.org/jira/browse/FLINK-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511131#comment-16511131
 ] 

ASF GitHub Bot commented on FLINK-9257:
---------------------------------------

Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/6053#discussion_r195079715
  
    --- Diff: flink-end-to-end-tests/test-scripts/test-runner-common.sh ---
    @@ -0,0 +1,76 @@
    +#!/usr/bin/env bash
    
+################################################################################
    +# Licensed to the Apache Software Foundation (ASF) under one
    +# or more contributor license agreements.  See the NOTICE file
    +# distributed with this work for additional information
    +# regarding copyright ownership.  The ASF licenses this file
    +# to you under the Apache License, Version 2.0 (the
    +# "License"); you may not use this file except in compliance
    +# with the License.  You may obtain a copy of the License at
    +#
    +#     http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    
+################################################################################
    +
    +source "${END_TO_END_DIR}"/test-scripts/common.sh
    +
    +#######################################
    +# Prints the given description, runs the given test and prints how long 
the execution took.
    +# Arguments:
    +#   $1: description of the test
    +#   $2: command to execute
    +#######################################
    +function run_test {
    +    description="$1"
    +    command="$2"
    +
    +    printf 
"\n==============================================================================\n"
    +    printf "Running '${description}'\n"
    +    printf 
"==============================================================================\n"
    +    start_timer
    +    ${command}
    +    exit_code="$?"
    +    time_elapsed=$(end_timer)
    +
    +    check_logs_for_errors
    +    check_logs_for_exceptions
    +    check_logs_for_non_empty_out_files
    +
    +    cleanup
    +
    +    if [[ ${exit_code} == 0 ]]; then
    +        if [[ ! "$PASS" ]]; then
    --- End diff --
    
    I thought we aren't using pass anymore?


> End-to-end tests prints "All tests PASS" even if individual test-script 
> returns non-zero exit code
> --------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-9257
>                 URL: https://issues.apache.org/jira/browse/FLINK-9257
>             Project: Flink
>          Issue Type: Bug
>          Components: Tests
>    Affects Versions: 1.5.0
>            Reporter: Florian Schmidt
>            Assignee: Florian Schmidt
>            Priority: Critical
>             Fix For: 1.6.0
>
>
> In some cases the test-suite exits with non-zero exit code but still prints 
> "All tests PASS" to stdout. This happens because how the test runner works, 
> which is roughly as follows
>  # Either run-nightly-tests.sh or run-precommit-tests.sh executes a suite of 
> tests consisting of one multiple bash scripts.
>  # As soon as one of those bash scripts exists with non-zero exit code, the 
> tests won't continue to run and the test-suite will also exit with non-zero 
> exit code.
>  # *During the cleanup hook (trap cleanup EXIT in common.sh) it will be 
> checked whether there are non-empty out files or log files with certain 
> exceptions. If a tests fails with non-zero exit code, but does not have any 
> exceptions or .out files, this will still print "All tests PASS" to stdout, 
> even though they don't*
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to