[
https://issues.apache.org/jira/browse/BEAM-10856?focusedWorklogId=515874&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515874
]
ASF GitHub Bot logged work on BEAM-10856:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Nov/20 03:16
Start Date: 24/Nov/20 03:16
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #12779:
URL: https://github.com/apache/beam/pull/12779#issuecomment-732610284
> > > BeamAssertException
> >
> >
> > This one looks like a test on the asserts.
> > The actual error (which is likely a flake) is:
> > logs: [#12779
(checks)](https://github.com/apache/beam/pull/12779/checks?check_run_id=1433935136)
> > ```
> > apache_beam\runners\interactive\recording_manager_test.py:75:
> > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _
> > apache_beam\runners\interactive\interactive_environment.py:118: in
new_env
> > _interactive_beam_env.cleanup()
> > apache_beam\runners\interactive\interactive_environment.py:272: in
cleanup
> > cache_manager.cleanup()
> > apache_beam\runners\interactive\caching\streaming_cache.py:391: in
cleanup
> > shutil.rmtree(self._cache_dir)
> > c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:500: in rmtree
> > return _rmtree_unsafe(path, onerror)
> > c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:390: in
_rmtree_unsafe
> > _rmtree_unsafe(fullname, onerror)
> > c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:395: in
_rmtree_unsafe
> > onerror(os.unlink, fullname, sys.exc_info())
> > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _
> >
> > ...
> >
> > > os.unlink(fullname)
> > E PermissionError: [WinError 32] The process cannot
access the file because it is being used by another process:
'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py36-win\\tmp\\it-4m1c1oje2145793178144\\full\\fb91a47796-2145832985040-2145832986608-2145793178144'
> > ```
> >
> >
> > Maybe clean should catch the error and not fail? Or try again?
> > @KevinGG - Could you check this error?
>
> The problem is windows specific. I tried on Mac OS, the `shutil.rmtree`
works even if the file handler is still open:
> <img alt="Screen Shot 2020-11-23 at 5 11 51 PM" width="964"
src="https://user-images.githubusercontent.com/4423149/100034273-b343ef80-2db0-11eb-8988-dcfc2c79322c.png";>
> It's said that on windows, if a file is opened, even if in the same
process deleting it (or its parent directory), you will see this error:
[stackoverflow.com/questions/27215462/permissionerror-winerror-32-the-process-cannot-access-the-file-because-it-is](https://stackoverflow.com/questions/27215462/permissionerror-winerror-32-the-process-cannot-access-the-file-because-it-is).
>
> Potential breaking point is in the streaming cache where the `open` and
`close` are explicitly invoked (where `with` is inappropriate to use):
> <img alt="Screen Shot 2020-11-23 at 5 44 53 PM" width="621"
src="https://user-images.githubusercontent.com/4423149/100035692-a7a5f800-2db3-11eb-8ec8-abc8923406c0.png";>
> Is it possible that a `finish_bundle` would fail silently or not executed
at all?
>
> This probably does not happen that often, we can revisit this if it
happens frequently and ignore this by specifying `shutil.rmtree(path,
ignore_errors=True, onerror=...)` since it's just failing to clean up useless
temp files.
Maybe file a JIRA and we can track how often this happens? I do not think
people triage all failures so it is hard to tell how frequently this happens in
pre commit tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 515874)
Time Spent: 9h (was: 8h 50m)
> Support NestedValueProvider for Python SDK
> ------------------------------------------
>
> Key: BEAM-10856
> URL: https://issues.apache.org/jira/browse/BEAM-10856
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Reporter: Ashwin Ramaswami
> Priority: P2
> Time Spent: 9h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
![https://user-images.githubusercontent.com/4423149/100034273-b343ef80-2db0-11eb-8988-dcfc2c79322c.png"](https://user-images.githubusercontent.com/4423149/100034273-b343ef80-2db0-11eb-8988-dcfc2c79322c.png"){kind=link}
![https://user-images.githubusercontent.com/4423149/100035692-a7a5f800-2db3-11eb-8ec8-abc8923406c0.png"](https://user-images.githubusercontent.com/4423149/100035692-a7a5f800-2db3-11eb-8ec8-abc8923406c0.png"){kind=link}