[
https://issues.apache.org/jira/browse/BEAM-3736?focusedWorklogId=507841&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-507841
]
ASF GitHub Bot logged work on BEAM-3736:
----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Nov/20 23:15
Start Date: 04/Nov/20 23:15
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #13048:
URL: https://github.com/apache/beam/pull/13048#issuecomment-722025568
I am seeing an error in one of the tests that we run internally with this
change:
```
test_global_fanout (apache_beam.transforms.combiners_test.CombineTest) ...
ERROR
======================================================================
ERROR: test_global_fanout (apache_beam.transforms.combiners_test.CombineTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/build/work/12791c230bd5926c2b95fef2c4fb1814e38e/google3/runfiles/google3/third_party/py/apache_beam/transforms/combiners_test.py",
line 544, in test_global_fanout
assert_that(result, equal_to([49.5]))
File
"/build/work/12791c230bd5926c2b95fef2c4fb1814e38e/google3/runfiles/google3/third_party/py/apache_beam/pipeline.py",
line 568, in __exit__
self.result = self.run()
File
"/build/work/12791c230bd5926c2b95fef2c4fb1814e38e/google3/runfiles/google3/third_party/py/apache_beam/testing/test_pipeline.py",
line 112, in run
False if self.not_use_test_runner_api else test_runner_api))
File
"/build/work/12791c230bd5926c2b95fef2c4fb1814e38e/google3/runfiles/google3/third_party/py/apache_beam/pipeline.py",
line 518, in run
allow_proto_holders=True).run(False)
File
"/build/work/12791c230bd5926c2b95fef2c4fb1814e38e/google3/runfiles/google3/third_party/py/apache_beam/pipeline.py",
line 547, in run
return self.runner.run_pipeline(self, self._options)
File
"/build/work/12791c230bd5926c2b95fef2c4fb1814e38e/google3/runfiles/google3/third_party/py/apache_beam/runners/dataflow/test_dataflow_runner.py",
line 57, in run_pipeline
self).run_pipeline(pipeline, options)
File
"/build/work/12791c230bd5926c2b95fef2c4fb1814e38e/google3/runfiles/google3/third_party/py/apache_beam/runners/dataflow/dataflow_runner.py",
line 495, in run_pipeline
self._check_for_unsupported_features_on_non_portable_worker(pipeline)
File
"/build/work/12791c230bd5926c2b95fef2c4fb1814e38e/google3/runfiles/google3/third_party/py/apache_beam/runners/dataflow/dataflow_runner.py",
line 467, in _check_for_unsupported_features_on_non_portable_worker
pipeline.visit(self.combinefn_visitor())
File
"/build/work/12791c230bd5926c2b95fef2c4fb1814e38e/google3/runfiles/google3/third_party/py/apache_beam/pipeline.py",
line 594, in visit
self._root_transform().visit(visitor, self, visited)
File
"/build/work/12791c230bd5926c2b95fef2c4fb1814e38e/google3/runfiles/google3/third_party/py/apache_beam/pipeline.py",
line 1085, in visit
part.visit(visitor, pipeline, visited)
File
"/build/work/12791c230bd5926c2b95fef2c4fb1814e38e/google3/runfiles/google3/third_party/py/apache_beam/pipeline.py",
line 1085, in visit
part.visit(visitor, pipeline, visited)
File
"/build/work/12791c230bd5926c2b95fef2c4fb1814e38e/google3/runfiles/google3/third_party/py/apache_beam/pipeline.py",
line 1085, in visit
part.visit(visitor, pipeline, visited)
[Previous line repeated 2 more times]
File
"/build/work/12791c230bd5926c2b95fef2c4fb1814e38e/google3/runfiles/google3/third_party/py/apache_beam/pipeline.py",
line 1088, in visit
visitor.visit_transform(self)
File
"/build/work/12791c230bd5926c2b95fef2c4fb1814e38e/google3/runfiles/google3/third_party/py/apache_beam/runners/dataflow/dataflow_runner.py",
line 430, in visit_transform
'CombineFn.setup and CombineFn.teardown are '
ValueError: CombineFn.setup and CombineFn.teardown are not supported with
non-portable Dataflow runner. Please use Dataflow Runner V2 instead.
----------------------------------------------------------------------
Ran 1 test in 1.216s
FAILED (errors=1)
```
I wonder if this broke ValidatesRunner tests on RunnerV1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 507841)
Time Spent: 10h 20m (was: 10h 10m)
> Add SetUp() and TearDown() for CombineFns
> -----------------------------------------
>
> Key: BEAM-3736
> URL: https://issues.apache.org/jira/browse/BEAM-3736
> Project: Beam
> Issue Type: Improvement
> Components: beam-model, sdk-py-core
> Reporter: Chuan Yu Foo
> Assignee: Kamil Wasilewski
> Priority: P3
> Time Spent: 10h 20m
> Remaining Estimate: 0h
>
> I have a CombineFn that has a large amount of state that needs to be loaded
> once before it can add_input or merge_combiners (for example, the CombineFn
> might load up a large lookup table used for combining).
> Right now, to initialise this state, for each of the methods, I check if the
> state has already been initialised, and if not, I initialise it. It would be
> nice if CombineFn provided a SetUp() method that is called once to initialise
> this state (and a corresponding TearDown() method to clean up this state if
> necessary).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
