[GitHub] [beam] KevinGG commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK

[GitBox]

KevinGG commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-732535245


   > > BeamAssertException
   > 
   > This one looks like a test on the asserts.
   > 
   > The actual error (which is likely a flake) is:
   > 
   > logs: 
https://github.com/apache/beam/pull/12779/checks?check_run_id=1433935136
   > 
   > ```
   > apache_beam\runners\interactive\recording_manager_test.py:75: 
   > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _
   > apache_beam\runners\interactive\interactive_environment.py:118: in new_env
   >     _interactive_beam_env.cleanup()
   > apache_beam\runners\interactive\interactive_environment.py:272: in cleanup
   >     cache_manager.cleanup()
   > apache_beam\runners\interactive\caching\streaming_cache.py:391: in cleanup
   >     shutil.rmtree(self._cache_dir)
   > c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:500: in rmtree
   >     return _rmtree_unsafe(path, onerror)
   > c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:390: in 
_rmtree_unsafe
   >     _rmtree_unsafe(fullname, onerror)
   > c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:395: in 
_rmtree_unsafe
   >     onerror(os.unlink, fullname, sys.exc_info())
   > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _
   > 
   > ...
   > 
   > >                   os.unlink(fullname)
   > E                   PermissionError: [WinError 32] The process cannot 
access the file because it is being used by another process: 
'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py36-win\\tmp\\it-4m1c1oje2145793178144\\full\\fb91a47796-2145832985040-2145832986608-2145793178144'
   > ```
   > 
   > Maybe clean should catch the error and not fail? Or try again?
   > 
   > @KevinGG - Could you check this error?
   
   
   The problem is windows specific. I tried on Mac OS, the `shutil.rmtree` 
works even if the file handler is still open:
   <img width="964" alt="Screen Shot 2020-11-23 at 5 11 51 PM" 
src="https://user-images.githubusercontent.com/4423149/100034273-b343ef80-2db0-11eb-8988-dcfc2c79322c.png";>
   It's said that on windows, if a file is opened, even if in the same process 
deleting it (or its parent directory), you will see this error: 
https://stackoverflow.com/questions/27215462/permissionerror-winerror-32-the-process-cannot-access-the-file-because-it-is.
   
   Potential breaking point is in the streaming cache where the `open` and 
`close` are explicitly invoked (where `with` is inappropriate to use):
   <img width="621" alt="Screen Shot 2020-11-23 at 5 44 53 PM" 
src="https://user-images.githubusercontent.com/4423149/100035692-a7a5f800-2db3-11eb-8ec8-abc8923406c0.png";>
   Is it possible that a `finish_bundle` would fail silently or not executed at 
all?
   
   This probably does not happen that often, we can revisit this if it happens 
frequently and ignore this by specifying `shutil.rmtree(path, 
ignore_errors=True, onerror=...)` since it's just failing to clean up useless 
temp files.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org