[
https://issues.apache.org/jira/browse/BEAM-9640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277627#comment-17277627
]
Pablo Estrada edited comment on BEAM-9640 at 2/3/21, 1:55 AM:
--------------------------------------------------------------
As of right now, most tests are passing (all tests pass if I remove assertion
checks for MIN and MAX watermarks before and after execution of stages).
Failing test:
{code:java}
FAIL: test_draining_sdf_with_sdf_initiated_checkpointing
(apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTest)
{code}
This test includes an SDF expansion with support for drain. The reason it
fails is that there is some sort of delayed application on
`ref_PCollection_PCollection_8_truncate_restriction`. I am investigating this.
Shape of the pipeline (main inputs in bold links):
!graph.png|width=893,height=436!
To investigate:
* The PR adding support for drain in Python:
[https://github.com/apache/beam/pull/12289/]
*
was (Author: pabloem):
As of right now, most tests are passing (all tests pass if I remove assertion
checks for MIN and MAX watermarks before and after execution of stages).
Failing test:
{code:java}
FAIL: test_draining_sdf_with_sdf_initiated_checkpointing
(apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTest)
{code}
This test includes an SDF expansion with support for drain. The reason it
fails is that there is some sort of delayed application on
`ref_PCollection_PCollection_8_truncate_restriction`. I am investigating this.
Shape of the pipeline:
!graph.png|width=893,height=436!
To investigate:
* The PR adding support for drain in Python:
[https://github.com/apache/beam/pull/12289/]
*
> Track PCollection watermark across bundle executions
> ----------------------------------------------------
>
> Key: BEAM-9640
> URL: https://issues.apache.org/jira/browse/BEAM-9640
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Pablo Estrada
> Priority: P3
> Attachments: graph.png
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> This can be done without relying on the watermark manager for execution.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
