[
https://issues.apache.org/jira/browse/BEAM-13040?focusedWorklogId=665464&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-665464
]
ASF GitHub Bot logged work on BEAM-13040:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Oct/21 00:36
Start Date: 14/Oct/21 00:36
Worklog Time Spent: 10m
Work Description: codecov[bot] edited a comment on pull request #15719:
URL: https://github.com/apache/beam/pull/15719#issuecomment-942828140
#
[Codecov](https://codecov.io/gh/apache/beam/pull/15719?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
Report
> Merging
[#15719](https://codecov.io/gh/apache/beam/pull/15719?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
(a563ddd) into
[master](https://codecov.io/gh/apache/beam/commit/f624c8059e610d1680fb869de739cdc3b831a6de?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
(f624c80) will **increase** coverage by `0.00%`.
> The diff coverage is `94.25%`.
[](https://codecov.io/gh/apache/beam/pull/15719?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15719 +/- ##
========================================
Coverage 83.49% 83.50%
========================================
Files 445 445
Lines 61415 61196 -219
========================================
- Hits 51281 51101 -180
+ Misses 10134 10095 -39
```
| [Impacted
Files](https://codecov.io/gh/apache/beam/pull/15719?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
| Coverage Δ | |
|---|---|---|
|
[...nners/portability/fn\_api\_runner/worker\_handlers.py](https://codecov.io/gh/apache/beam/pull/15719/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dvcmtlcl9oYW5kbGVycy5weQ==)
| `79.44% <ø> (+0.10%)` | :arrow_up: |
|
[sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/15719/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5)
| `95.81% <ø> (-0.03%)` | :arrow_down: |
|
[...ers/portability/fn\_api\_runner/watermark\_manager.py](https://codecov.io/gh/apache/beam/pull/15719/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3dhdGVybWFya19tYW5hZ2VyLnB5)
| `96.00% <75.00%> (ø)` | |
|
[...eam/runners/portability/fn\_api\_runner/execution.py](https://codecov.io/gh/apache/beam/pull/15719/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2V4ZWN1dGlvbi5weQ==)
| `93.34% <89.74%> (+1.39%)` | :arrow_up: |
|
[sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/15719/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5)
| `86.21% <100.00%> (ø)` | |
|
[...eam/runners/portability/fn\_api\_runner/fn\_runner.py](https://codecov.io/gh/apache/beam/pull/15719/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2ZuX3J1bm5lci5weQ==)
| `90.80% <100.00%> (+1.05%)` | :arrow_up: |
|
[.../runners/portability/fn\_api\_runner/translations.py](https://codecov.io/gh/apache/beam/pull/15719/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL3RyYW5zbGF0aW9ucy5weQ==)
| `93.20% <100.00%> (+0.09%)` | :arrow_up: |
|
[...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15719/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==)
| `88.70% <0.00%> (-0.46%)` | :arrow_down: |
|
[...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/15719/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=)
| `82.25% <0.00%> (-0.14%)` | :arrow_down: |
|
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15719/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
| `93.64% <0.00%> (+0.12%)` | :arrow_up: |
| ... and [7
more](https://codecov.io/gh/apache/beam/pull/15719/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
| |
------
[Continue to review full report at
Codecov](https://codecov.io/gh/apache/beam/pull/15719?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn
more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by
[Codecov](https://codecov.io/gh/apache/beam/pull/15719?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
Last update
[a9120e0...a563ddd](https://codecov.io/gh/apache/beam/pull/15719?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
Read the [comment
docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 665464)
Time Spent: 0.5h (was: 20m)
> AsIter side input is not correctly recognized as a dependency.
> --------------------------------------------------------------
>
> Key: BEAM-13040
> URL: https://issues.apache.org/jira/browse/BEAM-13040
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Environment: Linux Debian 5.10 x86_64
> Python 3.8.11
> Reporter: Jongbin Park
> Priority: P2
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The error is happening at current master (head). It is fine on the latest
> release (2.33.0).
> Example to reproduce:
> {code:python}
> import unittest
> import apache_beam as beam
> class LoggingFn(beam.DoFn):
> def __init__(self, name):
> self._name = name
> def process(self, element, *side_inputs):
> print(f'Running {self._name} (side inputs: {[list(s) for s in
> side_inputs]})')
> return [self._name]
> class BeamDagTest(unittest.TestCase):
> def test_dag(self):
> with beam.Pipeline() as p:
> root = p | 'CreateRoot' >> beam.Create([None])
> example_gen = root | 'CsvExampleGen' >> beam.ParDo(
> LoggingFn('CsvExampleGen'),
> )
> statistics_gen = root | 'StatisticsGen' >> beam.ParDo(
> LoggingFn('StatisticsGen'),
> beam.pvalue.AsIter(example_gen), # AsIter to specify upstream task
> dependency.
> )
> schema_gen = root | 'SchemaGen' >> beam.ParDo(
> LoggingFn('SchemaGen'),
> beam.pvalue.AsIter(statistics_gen),
> )
> example_validator = root | 'ExampleValidator' >> beam.ParDo(
> LoggingFn('ExampleValidator'),
> beam.pvalue.AsIter(statistics_gen),
> beam.pvalue.AsIter(schema_gen),
> )
> transform = root | 'Transform' >> beam.ParDo(
> LoggingFn('Transform'),
> beam.pvalue.AsIter(example_gen),
> beam.pvalue.AsIter(schema_gen),
> )
> trainer = root | 'Trainer' >> beam.ParDo(
> LoggingFn('Trainer'),
> beam.pvalue.AsIter(example_gen),
> beam.pvalue.AsIter(schema_gen),
> beam.pvalue.AsIter(transform),
> )
> model_resolver = root | 'latest_blessed_model_resolver' >> beam.ParDo(
> LoggingFn('latest_blessed_model_resolver'),
> )
> evaluator = root | 'Evaluator' >> beam.ParDo(
> LoggingFn('Evaluator'),
> beam.pvalue.AsIter(example_gen),
> beam.pvalue.AsIter(trainer),
> beam.pvalue.AsIter(model_resolver),
> )
> pusher = root | 'Pusher' >> beam.ParDo(
> LoggingFn('Pusher'),
> beam.pvalue.AsIter(trainer),
> beam.pvalue.AsIter(evaluator),
> ){code}
>
> According to AsIter
> [documentation|https://github.com/apache/beam/blob/64ec15fa2208d8f9b5ca5653866e1992fd07f7dc/sdks/python/apache_beam/pvalue.py#L527],
> entire PCollection should be made available as a side input, which means
> side input PTransform should run before the current PTransform. We used to
> exploit this feature to run DAG of tasks by injecting task dependency with
> side inputs, however this mechanism does not work properly in current master
> (71d7213d98):
> Output with apache-beam==2.33.0:
> {code:java}
> Running CsvExampleGen (side inputs: [])
> Running latest_blessed_model_resolver (side inputs: [])
> Running StatisticsGen (side inputs: [['CsvExampleGen']])
> Running SchemaGen (side inputs: [['StatisticsGen']])
> Running ExampleValidator (side inputs: [['StatisticsGen'], ['SchemaGen']])
> Running Transform (side inputs: [['CsvExampleGen'], ['SchemaGen']])
> Running Trainer (side inputs: [['CsvExampleGen'], ['SchemaGen'],
> ['Transform']])
> Running Evaluator (side inputs: [['CsvExampleGen'], ['Trainer'],
> ['latest_blessed_model_resolver']])
> Running Pusher (side inputs: [['Trainer'], ['Evaluator']]){code}
> Output with apache-beam installed from 71d7213d98 (origin/master):
> {code:java}
> Running CsvExampleGen (side inputs: [])
> Running latest_blessed_model_resolver (side inputs: [])
> Running StatisticsGen (side inputs: [['CsvExampleGen']])
> Running Pusher (side inputs: [[], []])
> Running Evaluator (side inputs: [['CsvExampleGen'], [],
> ['latest_blessed_model_resolver']])
> Running SchemaGen (side inputs: [['StatisticsGen']])
> Running Trainer (side inputs: [['CsvExampleGen'], ['SchemaGen'], []])
> Running ExampleValidator (side inputs: [['StatisticsGen'], ['SchemaGen']])
> Running Transform (side inputs: [['CsvExampleGen'], ['SchemaGen']]){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
