Valentyn Tymofieiev created BEAM-5492:
-----------------------------------------
Summary: Python Dataflow integration tests should export the
pipeline console output to Jenkins Test Result section
Key: BEAM-5492
URL: https://issues.apache.org/jira/browse/BEAM-5492
Project: Beam
Issue Type: New Feature
Components: testing
Reporter: Valentyn Tymofieiev
Assignee: Jason Kuster
I'm looking at a flake in a postcommit suite that happened last week in Python
Dataflow integration test (https://issues.apache.org/jira/browse/BEAM-5415),
and the existing logs are not sufficient to debug the problem. The test suite
failed, however:
1. Test Result section in Jenkins postcommit only includes unit tests, and does
not include any integration tests.
2. The only available information about the failure is buried in the Full
Console Log and is not very informative (copy-pasted below).
3. There is a line in the console log pointing to: "XML:
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml",
however as a non-committer I don't have access to Jenkins machines.
This makes rootcausing integration tests failures quite difficult, since we
cannot even reliably identify an ID of a failed Dataflow job from the test
output.
It would be helpful to capture and persist in Jenkins the invocation of
integration test, and all the console output that SDK produces during test
execution.
Sample available logs from console output.
13:41:54 test_leader_board_it
(apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ...
ERROR
13:41:54
13:41:54 ======================================================================
13:41:54 ERROR: test_leader_board_it
(apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)
13:41:54 ----------------------------------------------------------------------
13:41:54 Traceback (most recent call last):
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",
line 812, in run
13:41:54 test(orig)
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
line 45, in __call__
13:41:54 return self.run(*arg, **kwarg)
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
line 133, in run
13:41:54 self.runTest(result)
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
line 151, in runTest
13:41:54 test(result)
13:41:54 File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
13:41:54 return self.run(*args, **kwds)
13:41:54 File "/usr/lib/python2.7/unittest/case.py", line 329, in run
13:41:54 testMethod()
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py",
line 161, in test_leader_board_it
13:41:54 self.test_pipeline.get_full_options_as_args(**extra_opts))
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board.py",
line 345, in run
13:41:54 'total_score': 'INTEGER',
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
line 414, in __exit__
13:41:54 self.run().wait_until_finish()
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
line 394, in run
13:41:54 self.to_runner_api(), self.runner, self._options).run(False)
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
line 407, in run
13:41:54 return self.runner.run_pipeline(self)
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
line 68, in run_pipeline
13:41:54 self.result.cancel()
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
line 1167, in cancel
13:41:54 self._update_job()
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
line 1078, in _update_job
13:41:54 self._job = self._runner.dataflow_client.get_job(self.job_id())
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/utils/retry.py",
line 184, in wrapper
13:41:54 return fun(*args, **kwargs)
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",
line 629, in get_job
13:41:54 response = self._client.projects_locations_jobs.Get(request)
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py",
line 659, in Get
13:41:54 config, request, global_params=global_params)
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py",
line 720, in _RunMethod
13:41:54 http, http_request, **opts)
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 346, in MakeRequest
13:41:54 check_response_func=check_response_func)
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
line 396, in _MakeRequestNoRetry
13:41:54 redirections=redirections, connection_type=connection_type)
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py",
line 193, in new_request
13:41:54 redirections, connection_type)
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py",
line 282, in request
13:41:54 connection_type=connection_type)
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py",
line 1694, in request
13:41:54 (response, content) = self._request(conn, authority, uri,
request_uri, method, body, headers, redirections, cachekey)
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py",
line 1434, in _request
13:41:54 (response, content) = self._conn_request(conn, request_uri,
method, body, headers)
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py",
line 1390, in _conn_request
13:41:54 response = conn.getresponse()
13:41:54 File "/usr/lib/python2.7/httplib.py", line 1136, in getresponse
13:41:54 response.begin()
13:41:54 File "/usr/lib/python2.7/httplib.py", line 453, in begin
13:41:54 version, status, reason = self._read_status()
13:41:54 File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
13:41:54 line = self.fp.readline(_MAXLINE + 1)
13:41:54 File "/usr/lib/python2.7/socket.py", line 480, in readline
13:41:54 data = self._sock.recv(self._rbufsize)
13:41:54 File "/usr/lib/python2.7/ssl.py", line 756, in recv
13:41:54 return self.read(buflen)
13:41:54 File "/usr/lib/python2.7/ssl.py", line 643, in read
13:41:54 v = self._sslobj.read(len)
13:41:54 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",
line 276, in signalhandler
13:41:54 raise TimedOutException()
13:41:54 TimedOutException: 'test_leader_board_it
(apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)'
13:41:54
13:41:54 ----------------------------------------------------------------------
13:41:54 XML:
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml
13:41:54 ----------------------------------------------------------------------
13:41:54 Ran 14 tests in 3001.679s
13:41:54
13:41:54 FAILED (errors=1)
13:41:54
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
