[
https://issues.apache.org/jira/browse/BEAM-10921?focusedWorklogId=515691&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515691
]
ASF GitHub Bot logged work on BEAM-10921:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Nov/20 17:57
Start Date: 23/Nov/20 17:57
Worklog Time Spent: 10m
Work Description: davidyan74 commented on a change in pull request #13335:
URL: https://github.com/apache/beam/pull/13335#discussion_r528883529
##########
File path: sdks/python/apache_beam/pipeline.py
##########
@@ -504,10 +504,12 @@ def run(self, test_runner_api='AUTO'):
if test_runner_api == 'AUTO':
# Don't pay the cost of a round-trip if we're going to be going through
# the FnApi anyway...
+ # The InteractiveRunner relies a constant pipeline reference, skip it.
Review comment:
nit: relies -> relies on
##########
File path: sdks/python/apache_beam/pipeline.py
##########
@@ -504,10 +504,12 @@ def run(self, test_runner_api='AUTO'):
if test_runner_api == 'AUTO':
# Don't pay the cost of a round-trip if we're going to be going through
# the FnApi anyway...
+ # The InteractiveRunner relies a constant pipeline reference, skip it.
test_runner_api = (
not self.runner.is_fnapi_compatible() and (
self.runner.__class__.__name__ != 'SwitchingDirectRunner' or
- self._options.view_as(StandardOptions).streaming))
+ self._options.view_as(StandardOptions).streaming) and
+ self.runner.__class__.__name__ != 'InteractiveRunner')
Review comment:
Is there a better way for the runner to indicate it relies on a constant
pipeline reference?
##########
File path: sdks/python/apache_beam/runners/interactive/background_caching_job.py
##########
@@ -121,7 +121,8 @@ def state(self):
return self._pipeline_result.state
-def attempt_to_run_background_caching_job(runner, user_pipeline, options=None):
+def attempt_to_run_background_caching_job(
Review comment:
Should we take this opportunity to update the name according to the new
convention? e.g. "background caching job" -> "background source recording job".
It's fine if you plan to do this in a later PR instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 515691)
Time Spent: 6h 40m (was: 6.5h)
> Interactive Runner Python unit tests are flaky on Windows
> ---------------------------------------------------------
>
> Key: BEAM-10921
> URL: https://issues.apache.org/jira/browse/BEAM-10921
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Brian Hulette
> Assignee: Ning Kang
> Priority: P1
> Labels: currently-failing, flake
> Fix For: Not applicable
>
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> Over the past few days python unit tests have been failing frequently. The
> errors always seem to occur when cleaning up the interactive environment:
> {code}
> ...........
> [100%]
> ================================== FAILURES
> ===================================
> _
> PipelineInstrumentTest.test_able_to_cache_intermediate_unbounded_source_pcollection
> _
> [gw2] win32 -- Python 3.5.4
> d:\a\beam\beam\sdks\python\target\.tox\py35-win\scripts\python.exe
> self =
> <apache_beam.runners.interactive.pipeline_instrument_test.PipelineInstrumentTest
> testMethod=test_able_to_cache_intermediate_unbounded_source_pcollection>
> def setUp(self):
> > ie.new_env()
> apache_beam\runners\interactive\pipeline_instrument_test.py:46:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> apache_beam\runners\interactive\interactive_environment.py:117: in new_env
> _interactive_beam_env.cleanup()
> apache_beam\runners\interactive\interactive_environment.py:273: in cleanup
> cache_manager.cleanup()
> apache_beam\runners\interactive\caching\streaming_cache.py:391: in cleanup
> shutil.rmtree(self._cache_dir)
> c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:494: in rmtree
> return _rmtree_unsafe(path, onerror)
> c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:384: in
> _rmtree_unsafe
> _rmtree_unsafe(fullname, onerror)
> c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:389: in
> _rmtree_unsafe
> onerror(os.unlink, fullname, sys.exc_info())
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> path =
> 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py35-win\\tmp\\it-8vh2z7pi2021914046928\\full'
> onerror = <function rmtree.<locals>.onerror at 0x000001D6C3E5C7B8>
> def _rmtree_unsafe(path, onerror):
> try:
> if os.path.islink(path):
> # symlinks to directories are forbidden, see bug #1669
> raise OSError("Cannot call rmtree on a symbolic link")
> except OSError:
> onerror(os.path.islink, path, sys.exc_info())
> # can't continue even if onerror hook returns
> return
> names = []
> try:
> names = os.listdir(path)
> except OSError:
> onerror(os.listdir, path, sys.exc_info())
> for name in names:
> fullname = os.path.join(path, name)
> try:
> mode = os.lstat(fullname).st_mode
> except OSError:
> mode = 0
> if stat.S_ISDIR(mode):
> _rmtree_unsafe(fullname, onerror)
> else:
> try:
> > os.unlink(fullname)
> E PermissionError: [WinError 32] The process cannot access
> the file because it is being used by another process:
> 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py35-win\\tmp\\it-8vh2z7pi2021914046928\\full\\ac8879590f-2021876280456-2021876278608-2021914046928'
> c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:387: PermissionError
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
