[
https://issues.apache.org/jira/browse/BEAM-5492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-5492:
-------------------------------------
Assignee: Andoni Guzman
> Python Dataflow integration tests should export the pipeline console output
> to Jenkins Test Result section
> ----------------------------------------------------------------------------------------------------------
>
> Key: BEAM-5492
> URL: https://issues.apache.org/jira/browse/BEAM-5492
> Project: Beam
> Issue Type: New Feature
> Components: testing
> Reporter: Valentyn Tymofieiev
> Assignee: Andoni Guzman
> Priority: P3
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> I'm looking at a flake in a postcommit suite that happened last week in
> Python Dataflow integration test
> (https://issues.apache.org/jira/browse/BEAM-5415), and the existing logs are
> not sufficient to debug the problem. The test suite failed, however:
> 1. Test Result section in Jenkins postcommit only includes unit tests, and
> does not include any integration tests.
> 2. The only available information about the failure is buried in the Full
> Console Log and is not very informative (copy-pasted below).
> 3. There is a line in the console log pointing to: "XML:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml",
> however as a non-committer I don't have access to Jenkins machines.
> This makes rootcausing integration tests failures quite difficult, since we
> cannot even reliably identify an ID of a failed Dataflow job from the test
> output.
> It would be helpful to capture and persist in Jenkins the invocation of
> integration test, and all the console output that SDK produces during test
> execution.
> Sample available logs from console output.
> 13:41:54 test_leader_board_it
> (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ...
> ERROR
> 13:41:54
> 13:41:54
> ======================================================================
> 13:41:54 ERROR: test_leader_board_it
> (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)
> 13:41:54
> ----------------------------------------------------------------------
> 13:41:54 Traceback (most recent call last):
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",
> line 812, in run
> 13:41:54 test(orig)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
> line 45, in __call__
> 13:41:54 return self.run(*arg, **kwarg)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
> line 133, in run
> 13:41:54 self.runTest(result)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
> line 151, in runTest
> 13:41:54 test(result)
> 13:41:54 File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
> 13:41:54 return self.run(*args, **kwds)
> 13:41:54 File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> 13:41:54 testMethod()
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py",
> line 161, in test_leader_board_it
> 13:41:54 self.test_pipeline.get_full_options_as_args(**extra_opts))
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board.py",
> line 345, in run
> 13:41:54 'total_score': 'INTEGER',
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
> line 414, in __exit__
> 13:41:54 self.run().wait_until_finish()
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
> line 394, in run
> 13:41:54 self.to_runner_api(), self.runner, self._options).run(False)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
> line 407, in run
> 13:41:54 return self.runner.run_pipeline(self)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
> line 68, in run_pipeline
> 13:41:54 self.result.cancel()
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
> line 1167, in cancel
> 13:41:54 self._update_job()
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
> line 1078, in _update_job
> 13:41:54 self._job = self._runner.dataflow_client.get_job(self.job_id())
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/utils/retry.py",
> line 184, in wrapper
> 13:41:54 return fun(*args, **kwargs)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",
> line 629, in get_job
> 13:41:54 response = self._client.projects_locations_jobs.Get(request)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py",
> line 659, in Get
> 13:41:54 config, request, global_params=global_params)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py",
> line 720, in _RunMethod
> 13:41:54 http, http_request, **opts)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
> line 346, in MakeRequest
> 13:41:54 check_response_func=check_response_func)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
> line 396, in _MakeRequestNoRetry
> 13:41:54 redirections=redirections, connection_type=connection_type)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py",
> line 193, in new_request
> 13:41:54 redirections, connection_type)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py",
> line 282, in request
> 13:41:54 connection_type=connection_type)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py",
> line 1694, in request
> 13:41:54 (response, content) = self._request(conn, authority, uri,
> request_uri, method, body, headers, redirections, cachekey)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py",
> line 1434, in _request
> 13:41:54 (response, content) = self._conn_request(conn, request_uri,
> method, body, headers)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py",
> line 1390, in _conn_request
> 13:41:54 response = conn.getresponse()
> 13:41:54 File "/usr/lib/python2.7/httplib.py", line 1136, in getresponse
> 13:41:54 response.begin()
> 13:41:54 File "/usr/lib/python2.7/httplib.py", line 453, in begin
> 13:41:54 version, status, reason = self._read_status()
> 13:41:54 File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
> 13:41:54 line = self.fp.readline(_MAXLINE + 1)
> 13:41:54 File "/usr/lib/python2.7/socket.py", line 480, in readline
> 13:41:54 data = self._sock.recv(self._rbufsize)
> 13:41:54 File "/usr/lib/python2.7/ssl.py", line 756, in recv
> 13:41:54 return self.read(buflen)
> 13:41:54 File "/usr/lib/python2.7/ssl.py", line 643, in read
> 13:41:54 v = self._sslobj.read(len)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",
> line 276, in signalhandler
> 13:41:54 raise TimedOutException()
> 13:41:54 TimedOutException: 'test_leader_board_it
> (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)'
> 13:41:54
> 13:41:54
> ----------------------------------------------------------------------
> 13:41:54 XML:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml
> 13:41:54
> ----------------------------------------------------------------------
> 13:41:54 Ran 14 tests in 3001.679s
> 13:41:54
> 13:41:54 FAILED (errors=1)
> 13:41:54
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
