[
https://issues.apache.org/jira/browse/BEAM-14146?focusedWorklogId=768219&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-768219
]
ASF GitHub Bot logged work on BEAM-14146:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 09/May/22 23:22
Start Date: 09/May/22 23:22
Worklog Time Spent: 10m
Work Description: codecov[bot] commented on PR #17595:
URL: https://github.com/apache/beam/pull/17595#issuecomment-1121680405
#
[Codecov](https://codecov.io/gh/apache/beam/pull/17595?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
Report
> Merging
[#17595](https://codecov.io/gh/apache/beam/pull/17595?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
(748b90b) into
[release-2.39.0](https://codecov.io/gh/apache/beam/commit/8aa77e61e5ffc1efdc07772c591d4ab0a336533d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
(8aa77e6) will **increase** coverage by `0.03%`.
> The diff coverage is `100.00%`.
```diff
@@ Coverage Diff @@
## release-2.39.0 #17595 +/- ##
==================================================
+ Coverage 73.83% 73.86% +0.03%
==================================================
Files 691 691
Lines 91249 91533 +284
==================================================
+ Hits 67371 67614 +243
- Misses 22650 22691 +41
Partials 1228 1228
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `83.68% <100.00%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click
here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment)
to find out more.
| [Impacted
Files](https://codecov.io/gh/apache/beam/pull/17595?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
| Coverage Δ | |
|---|---|---|
|
[sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/17595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5)
| `83.15% <100.00%> (+0.17%)` | :arrow_up: |
|
[sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/17595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==)
| `94.49% <0.00%> (-2.65%)` | :arrow_down: |
|
[sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/17595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=)
| `87.94% <0.00%> (-2.33%)` | :arrow_down: |
|
[...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/17595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==)
| `87.50% <0.00%> (-1.71%)` | :arrow_down: |
|
[sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/17595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=)
| `88.01% <0.00%> (-1.39%)` | :arrow_down: |
|
[sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/17595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5)
| `90.97% <0.00%> (-0.76%)` | :arrow_down: |
|
[...eam/runners/portability/fn\_api\_runner/execution.py](https://codecov.io/gh/apache/beam/pull/17595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2V4ZWN1dGlvbi5weQ==)
| `92.44% <0.00%> (-0.65%)` | :arrow_down: |
|
[...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/17595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==)
| `88.90% <0.00%> (-0.64%)` | :arrow_down: |
|
[sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/17595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=)
| `93.43% <0.00%> (+0.01%)` | :arrow_up: |
|
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/17595/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
| `93.68% <0.00%> (+0.03%)` | :arrow_up: |
| ... and [9
more](https://codecov.io/gh/apache/beam/pull/17595/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
| |
------
[Continue to review full report at
Codecov](https://codecov.io/gh/apache/beam/pull/17595?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn
more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by
[Codecov](https://codecov.io/gh/apache/beam/pull/17595?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
Last update
[8aa77e6...748b90b](https://codecov.io/gh/apache/beam/pull/17595?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
Read the [comment
docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
Issue Time Tracking
-------------------
Worklog Id: (was: 768219)
Time Spent: 1.5h (was: 1h 20m)
> Python Streaming job failing to drain with BigQueryIO write errors
> ------------------------------------------------------------------
>
> Key: BEAM-14146
> URL: https://issues.apache.org/jira/browse/BEAM-14146
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp, sdk-py-core
> Affects Versions: 2.37.0
> Reporter: Rahul Iyer
> Assignee: Heejong Lee
> Priority: P1
> Fix For: 2.39.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> We have a Python Streaming Dataflow job that writes to BigQuery using the
> {{FILE_LOADS}} method and {{auto_sharding}} enabled. When we try to drain the
> job it fails with the following error,
> {code:python}
> "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery_tools.py",
> line 1000, in perform_load_job ValueError: Either a non-empty list of
> fully-qualified source URIs must be provided via the source_uris parameter or
> an open file object must be provided via the source_stream parameter.
> {code}
> Our {{WriteToBigQuery}} configuration,
> {code:python}
> beam.io.WriteToBigQuery(
> table=options.output_table,
> schema=bq_schema,
> create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
> write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND,
> insert_retry_strategy=RetryStrategy.RETRY_ON_TRANSIENT_ERROR,
> method=beam.io.WriteToBigQuery.Method.FILE_LOADS,
> additional_bq_parameters={
> "timePartitioning": {
> "type": "HOUR",
> "field": "bq_insert_timestamp",
> },
> "schemaUpdateOptions": ["ALLOW_FIELD_ADDITION", "ALLOW_FIELD_RELAXATION"],
> },
> triggering_frequency=120,
> with_auto_sharding=True,
> )
> {code}
> We are also noticing that the job only fails to drain when there are actual
> schema updates. If there are no schema updates the job drains without the
> above error.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
