That exit code bit is likely a red herring. And WAI. I recently changed the script to have a debug header that makes the execution clearer and gradle doesn't simply swallow the test output Part of it echos executing lines with the shell variables subbed in, because that's more useful for debugging.
In this case the '$?' is getting it's current value which is from the previous debugging call (that checks if the printout would have non empty output). It is confusing though so we should probably omit that line. Regardless we can be sure the error code was in fact 1 in that test run. On Mon, Oct 4, 2021, 6:18 PM Ke Wu <[email protected]> wrote: > Thanks for looking into this! > > I am curious why `TEST_EXIT_CODE` is logged to be 0 if there are actually > test failures, I did not looking into any test failures because of it. Do > you happen to know if this is expected or not? > > Best, > Ke > > On Oct 4, 2021, at 5:43 PM, Robert Burke <[email protected]> wrote: > > Looking at it now that looks like it covers the failures for samza (it's > passing). So we can merge that in if you'll review it. > > On Mon, Oct 4, 2021, 5:41 PM Robert Burke <[email protected]> wrote: > >> I was looking into these too. Some unfiltered out tests are failing (like >> a newly added check for post job metrics, is getting back all 0s from >> Samza). >> >> In the linked case, one of the Cross Language tests failed. (Search for >> "--- F" failing test runs. >> >> I have a PR that I'm using to iterate on and disabled the failing tests >> https://github.com/apache/beam/pull/15659 >> >> But Jenkins had not started execution after 16 minutes so i called it a >> day. >> >> On Mon, Oct 4, 2021, 5:30 PM Ke Wu <[email protected]> wrote: >> >>> Hello All, >>> >>> Samza Go Validates Runner started fail around 10/01/2021[1], the three >>> commits that contributes to the first failure [2] looks innocent, I also >>> tried to revert them but still got the same failure. >>> >>> What is confusing is the console log: >>> >>> *11:15:43* --- PASS: TestEmitParDoAfterGBK (5.70s)*11:15:43* PASS*11:15:43* >>> ok github.com/apache/beam/sdks/v2/go/test/regression >>> 38.517s*11:15:43* $ TEST_EXIT_CODE=0*11:15:43* $ cd ../..*11:15:43* $ exit >>> 1*11:15:43* $ exit 1 >>> >>> >>> Which suggests that tests all passed and expected to exit with code 0, >>> however, for some reason, the go script exited 1 instead [3], however, >>> reading from the script, if TEST_EXIT_CODE printed out to be 0, then it is >>> supposed to be exited 0. >>> >>> Does anyone have idea why this happens? >>> >>> Best, >>> Ke >>> >>> >>> [1] https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/ >>> [2] https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/319/ >>> [3] >>> https://github.com/apache/beam/blob/master/sdks/go/test/run_validatesrunner_tests.sh#L399 >>> >>> >> >
