[
https://issues.apache.org/jira/browse/BEAM-10921?focusedWorklogId=516333&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516333
]
ASF GitHub Bot logged work on BEAM-10921:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Nov/20 20:23
Start Date: 24/Nov/20 20:23
Worklog Time Spent: 10m
Work Description: rohdesamuel commented on a change in pull request
#13335:
URL: https://github.com/apache/beam/pull/13335#discussion_r529855458
##########
File path: sdks/python/apache_beam/pipeline.py
##########
@@ -504,10 +504,12 @@ def run(self, test_runner_api='AUTO'):
if test_runner_api == 'AUTO':
# Don't pay the cost of a round-trip if we're going to be going through
# the FnApi anyway...
+ # The InteractiveRunner relies a constant pipeline reference, skip it.
test_runner_api = (
not self.runner.is_fnapi_compatible() and (
self.runner.__class__.__name__ != 'SwitchingDirectRunner' or
- self._options.view_as(StandardOptions).streaming))
+ self._options.view_as(StandardOptions).streaming) and
+ self.runner.__class__.__name__ != 'InteractiveRunner')
Review comment:
@TheNeuralBit what do you think? I was following the precedence of the
SwitchingDirectRunner. The InteractiveRunner already does this check in its
code in multiple places, so there wouldn't be a safety regression if this line
is added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 516333)
Time Spent: 7h 10m (was: 7h)
> Interactive Runner Python unit tests are flaky on Windows
> ---------------------------------------------------------
>
> Key: BEAM-10921
> URL: https://issues.apache.org/jira/browse/BEAM-10921
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Brian Hulette
> Assignee: Ning Kang
> Priority: P1
> Labels: currently-failing, flake
> Fix For: Not applicable
>
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> Over the past few days python unit tests have been failing frequently. The
> errors always seem to occur when cleaning up the interactive environment:
> {code}
> ...........
> [100%]
> ================================== FAILURES
> ===================================
> _
> PipelineInstrumentTest.test_able_to_cache_intermediate_unbounded_source_pcollection
> _
> [gw2] win32 -- Python 3.5.4
> d:\a\beam\beam\sdks\python\target\.tox\py35-win\scripts\python.exe
> self =
> <apache_beam.runners.interactive.pipeline_instrument_test.PipelineInstrumentTest
> testMethod=test_able_to_cache_intermediate_unbounded_source_pcollection>
> def setUp(self):
> > ie.new_env()
> apache_beam\runners\interactive\pipeline_instrument_test.py:46:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> apache_beam\runners\interactive\interactive_environment.py:117: in new_env
> _interactive_beam_env.cleanup()
> apache_beam\runners\interactive\interactive_environment.py:273: in cleanup
> cache_manager.cleanup()
> apache_beam\runners\interactive\caching\streaming_cache.py:391: in cleanup
> shutil.rmtree(self._cache_dir)
> c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:494: in rmtree
> return _rmtree_unsafe(path, onerror)
> c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:384: in
> _rmtree_unsafe
> _rmtree_unsafe(fullname, onerror)
> c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:389: in
> _rmtree_unsafe
> onerror(os.unlink, fullname, sys.exc_info())
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> path =
> 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py35-win\\tmp\\it-8vh2z7pi2021914046928\\full'
> onerror = <function rmtree.<locals>.onerror at 0x000001D6C3E5C7B8>
> def _rmtree_unsafe(path, onerror):
> try:
> if os.path.islink(path):
> # symlinks to directories are forbidden, see bug #1669
> raise OSError("Cannot call rmtree on a symbolic link")
> except OSError:
> onerror(os.path.islink, path, sys.exc_info())
> # can't continue even if onerror hook returns
> return
> names = []
> try:
> names = os.listdir(path)
> except OSError:
> onerror(os.listdir, path, sys.exc_info())
> for name in names:
> fullname = os.path.join(path, name)
> try:
> mode = os.lstat(fullname).st_mode
> except OSError:
> mode = 0
> if stat.S_ISDIR(mode):
> _rmtree_unsafe(fullname, onerror)
> else:
> try:
> > os.unlink(fullname)
> E PermissionError: [WinError 32] The process cannot access
> the file because it is being used by another process:
> 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py35-win\\tmp\\it-8vh2z7pi2021914046928\\full\\ac8879590f-2021876280456-2021876278608-2021914046928'
> c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:387: PermissionError
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
