[
https://issues.apache.org/jira/browse/BEAM-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322524#comment-17322524
]
Ning Kang commented on BEAM-12178:
----------------------------------
I think the root cause is in windows, shutil.rmtree cannot delete a directory
with files that are open by other processes.
In StreamingCache, there is a place where "open"/"close" a file is decoupled:
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/caching/streaming_cache.py#L116.
This test is not flaky. The flakiness is the cleanup of the current interactive
environment after last test execution. Some of the opened file handlers are not
closed.
I'll do some further investigation.
> ReadCacheTest flakes on Windows
> -------------------------------
>
> Key: BEAM-12178
> URL: https://issues.apache.org/jira/browse/BEAM-12178
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core, test-failures
> Reporter: Brian Hulette
> Assignee: Ning Kang
> Priority: P1
> Labels: currently-failing, flake
>
> Example failure:
> https://github.com/apache/beam/pull/14382/checks?check_run_id=2325304757
> {code}
> ________________________ ReadCacheTest.test_read_cache
> ________________________
> [gw4] win32 -- Python 3.6.8
> d:\a\beam\beam\sdks\python\target\.tox\py36-win\scripts\python.exe
> self = <apache_beam.runners.interactive.caching.read_cache_test.ReadCacheTest
> testMethod=test_read_cache>
> def setUp(self):
> > ie.new_env()
> apache_beam\runners\interactive\caching\read_cache_test.py:37:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> apache_beam\runners\interactive\interactive_environment.py:119: in new_env
> _interactive_beam_env.cleanup()
> apache_beam\runners\interactive\interactive_environment.py:269: in cleanup
> self.evict_recording_manager(pipeline)
> apache_beam\runners\interactive\interactive_environment.py:396: in
> evict_recording_manager
> rm.clear()
> apache_beam\runners\interactive\recording_manager.py:339: in clear
> cache_manager.cleanup()
> apache_beam\runners\interactive\caching\streaming_cache.py:385: in cleanup
> shutil.rmtree(self._cache_dir)
> c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:500: in rmtree
> return _rmtree_unsafe(path, onerror)
> c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:390: in
> _rmtree_unsafe
> _rmtree_unsafe(fullname, onerror)
> c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:395: in
> _rmtree_unsafe
> onerror(os.unlink, fullname, sys.exc_info())
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> path =
> 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py36-win\\tmp\\it-3ggiapgo2494199486000\\full'
> onerror = <function rmtree.<locals>.onerror at 0x00000244BA770A60>
> def _rmtree_unsafe(path, onerror):
> try:
> if os.path.islink(path):
> # symlinks to directories are forbidden, see bug #1669
> raise OSError("Cannot call rmtree on a symbolic link")
> except OSError:
> onerror(os.path.islink, path, sys.exc_info())
> # can't continue even if onerror hook returns
> return
> names = []
> try:
> names = os.listdir(path)
> except OSError:
> onerror(os.listdir, path, sys.exc_info())
> for name in names:
> fullname = os.path.join(path, name)
> try:
> mode = os.lstat(fullname).st_mode
> except OSError:
> mode = 0
> if stat.S_ISDIR(mode):
> _rmtree_unsafe(fullname, onerror)
> else:
> try:
> > os.unlink(fullname)
> E PermissionError: [WinError 32] The process cannot access
> the file because it is being used by another process:
> 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py36-win\\tmp\\it-3ggiapgo2494199486000\\full\\dacc5c76b6-2494199456376-2494201991240-2494199486000'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
