[
https://issues.apache.org/jira/browse/BEAM-5492?focusedWorklogId=765222&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-765222
]
ASF GitHub Bot logged work on BEAM-5492:
----------------------------------------
Author: ASF GitHub Bot
Created on: 03/May/22 00:01
Start Date: 03/May/22 00:01
Worklog Time Spent: 10m
Work Description: codecov[bot] commented on PR #17530:
URL: https://github.com/apache/beam/pull/17530#issuecomment-1115479934
#
[Codecov](https://codecov.io/gh/apache/beam/pull/17530?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
Report
> Merging
[#17530](https://codecov.io/gh/apache/beam/pull/17530?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
(f7403ed) into
[master](https://codecov.io/gh/apache/beam/commit/12330d8cffc06a067c4f26c8733663e0d7fcbdc1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
(12330d8) will **increase** coverage by `9.07%`.
> The diff coverage is `63.63%`.
```diff
@@ Coverage Diff @@
## master #17530 +/- ##
==========================================
+ Coverage 74.61% 83.68% +9.07%
==========================================
Files 653 464 -189
Lines 81802 64467 -17335
==========================================
- Hits 61037 53952 -7085
+ Misses 19784 10515 -9269
+ Partials 981 0 -981
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `83.68% <63.63%> (?)` | |
Flags with carried forward coverage won't be shown. [Click
here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment)
to find out more.
| [Impacted
Files](https://codecov.io/gh/apache/beam/pull/17530?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
| Coverage Δ | |
|---|---|---|
|
[...ache\_beam/runners/dataflow/test\_dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/17530/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy90ZXN0X2RhdGFmbG93X3J1bm5lci5weQ==)
| `31.48% <20.00%> (-1.18%)` | :arrow_down: |
|
[...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/17530/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=)
| `83.00% <100.00%> (+0.63%)` | :arrow_up: |
|
[sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/17530/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=)
| `87.80% <0.00%> (-7.32%)` | :arrow_down: |
|
[sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/17530/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==)
| `78.26% <0.00%> (-6.74%)` | :arrow_down: |
|
[sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/17530/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=)
| `84.89% <0.00%> (-5.82%)` | :arrow_down: |
|
[.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/17530/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5)
| `88.09% <0.00%> (-4.93%)` | :arrow_down: |
|
[sdks/python/apache\_beam/io/gcp/bigtableio.py](https://codecov.io/gh/apache/beam/pull/17530/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3RhYmxlaW8ucHk=)
| `74.69% <0.00%> (-4.79%)` | :arrow_down: |
|
[...n/apache\_beam/runners/interactive/cache\_manager.py](https://codecov.io/gh/apache/beam/pull/17530/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoZV9tYW5hZ2VyLnB5)
| `87.58% <0.00%> (-3.40%)` | :arrow_down: |
|
[...pache\_beam/dataframe/pandas\_top\_level\_functions.py](https://codecov.io/gh/apache/beam/pull/17530/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3BhbmRhc190b3BfbGV2ZWxfZnVuY3Rpb25zLnB5)
| `90.52% <0.00%> (-3.16%)` | :arrow_down: |
|
[...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/17530/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=)
| `89.43% <0.00%> (-3.16%)` | :arrow_down: |
| ... and [375
more](https://codecov.io/gh/apache/beam/pull/17530/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
| |
------
[Continue to review full report at
Codecov](https://codecov.io/gh/apache/beam/pull/17530?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn
more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by
[Codecov](https://codecov.io/gh/apache/beam/pull/17530?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
Last update
[12330d8...f7403ed](https://codecov.io/gh/apache/beam/pull/17530?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
Read the [comment
docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
Issue Time Tracking
-------------------
Worklog Id: (was: 765222)
Time Spent: 3.5h (was: 3h 20m)
> Python Dataflow integration tests should export the pipeline console output
> to Jenkins Test Result section
> ----------------------------------------------------------------------------------------------------------
>
> Key: BEAM-5492
> URL: https://issues.apache.org/jira/browse/BEAM-5492
> Project: Beam
> Issue Type: New Feature
> Components: testing
> Reporter: Valentyn Tymofieiev
> Priority: P3
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> I'm looking at a flake in a postcommit suite that happened last week in
> Python Dataflow integration test
> (https://issues.apache.org/jira/browse/BEAM-5415), and the existing logs are
> not sufficient to debug the problem. The test suite failed, however:
> 1. Test Result section in Jenkins postcommit only includes unit tests, and
> does not include any integration tests.
> 2. The only available information about the failure is buried in the Full
> Console Log and is not very informative (copy-pasted below).
> 3. There is a line in the console log pointing to: "XML:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml",
> however as a non-committer I don't have access to Jenkins machines.
> This makes rootcausing integration tests failures quite difficult, since we
> cannot even reliably identify an ID of a failed Dataflow job from the test
> output.
> It would be helpful to capture and persist in Jenkins the invocation of
> integration test, and all the console output that SDK produces during test
> execution.
> Sample available logs from console output.
> 13:41:54 test_leader_board_it
> (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ...
> ERROR
> 13:41:54
> 13:41:54
> ======================================================================
> 13:41:54 ERROR: test_leader_board_it
> (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)
> 13:41:54
> ----------------------------------------------------------------------
> 13:41:54 Traceback (most recent call last):
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",
> line 812, in run
> 13:41:54 test(orig)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
> line 45, in __call__
> 13:41:54 return self.run(*arg, **kwarg)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
> line 133, in run
> 13:41:54 self.runTest(result)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
> line 151, in runTest
> 13:41:54 test(result)
> 13:41:54 File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
> 13:41:54 return self.run(*args, **kwds)
> 13:41:54 File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> 13:41:54 testMethod()
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py",
> line 161, in test_leader_board_it
> 13:41:54 self.test_pipeline.get_full_options_as_args(**extra_opts))
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board.py",
> line 345, in run
> 13:41:54 'total_score': 'INTEGER',
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
> line 414, in __exit__
> 13:41:54 self.run().wait_until_finish()
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
> line 394, in run
> 13:41:54 self.to_runner_api(), self.runner, self._options).run(False)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
> line 407, in run
> 13:41:54 return self.runner.run_pipeline(self)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
> line 68, in run_pipeline
> 13:41:54 self.result.cancel()
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
> line 1167, in cancel
> 13:41:54 self._update_job()
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
> line 1078, in _update_job
> 13:41:54 self._job = self._runner.dataflow_client.get_job(self.job_id())
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/utils/retry.py",
> line 184, in wrapper
> 13:41:54 return fun(*args, **kwargs)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",
> line 629, in get_job
> 13:41:54 response = self._client.projects_locations_jobs.Get(request)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py",
> line 659, in Get
> 13:41:54 config, request, global_params=global_params)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py",
> line 720, in _RunMethod
> 13:41:54 http, http_request, **opts)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
> line 346, in MakeRequest
> 13:41:54 check_response_func=check_response_func)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
> line 396, in _MakeRequestNoRetry
> 13:41:54 redirections=redirections, connection_type=connection_type)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py",
> line 193, in new_request
> 13:41:54 redirections, connection_type)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py",
> line 282, in request
> 13:41:54 connection_type=connection_type)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py",
> line 1694, in request
> 13:41:54 (response, content) = self._request(conn, authority, uri,
> request_uri, method, body, headers, redirections, cachekey)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py",
> line 1434, in _request
> 13:41:54 (response, content) = self._conn_request(conn, request_uri,
> method, body, headers)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py",
> line 1390, in _conn_request
> 13:41:54 response = conn.getresponse()
> 13:41:54 File "/usr/lib/python2.7/httplib.py", line 1136, in getresponse
> 13:41:54 response.begin()
> 13:41:54 File "/usr/lib/python2.7/httplib.py", line 453, in begin
> 13:41:54 version, status, reason = self._read_status()
> 13:41:54 File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
> 13:41:54 line = self.fp.readline(_MAXLINE + 1)
> 13:41:54 File "/usr/lib/python2.7/socket.py", line 480, in readline
> 13:41:54 data = self._sock.recv(self._rbufsize)
> 13:41:54 File "/usr/lib/python2.7/ssl.py", line 756, in recv
> 13:41:54 return self.read(buflen)
> 13:41:54 File "/usr/lib/python2.7/ssl.py", line 643, in read
> 13:41:54 v = self._sslobj.read(len)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",
> line 276, in signalhandler
> 13:41:54 raise TimedOutException()
> 13:41:54 TimedOutException: 'test_leader_board_it
> (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)'
> 13:41:54
> 13:41:54
> ----------------------------------------------------------------------
> 13:41:54 XML:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml
> 13:41:54
> ----------------------------------------------------------------------
> 13:41:54 Ran 14 tests in 3001.679s
> 13:41:54
> 13:41:54 FAILED (errors=1)
> 13:41:54
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
