Github user heary-cao commented on the issue:
https://github.com/apache/spark/pull/17869
@HyukjinKwon
this changes before and after deleting files log information:
**Before changes**
> Run the unit test: SortShuffleManager properly cleans up files for
shuffles that use the serialized path
```
remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\blockmgr-9c553fa3-dfc9-435d-8624-ab48961c8248\0c
remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\blockmgr-9c553fa3-dfc9-435d-8624-ab48961c8248\0d
...
remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\blockmgr-9c553fa3-dfc9-435d-8624-ab48961c8248
remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\spark-c0db2905-02f1-4bbb-811b-3cf910dee7cf\userFiles-d8906c18-17ab-41e7-917e-11cd7c71243e
remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\spark-c0db2905-02f1-4bbb-811b-3cf910dee7cf
remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a
```
> Run the unit test: SortShuffleManager properly cleans up files for
shuffles that use the deserialized path
```
remove:C:\Users\Default\AppData\Local\Temp\spark-371eb96b-738f-4f6f-97c9-7a694afc34cc\blockmgr-0691abbf-61b8-41eb-9476-92479d9209d2\0c
remove:C:\Users\Default\AppData\Local\Temp\spark-371eb96b-738f-4f6f-97c9-7a694afc34cc\blockmgr-0691abbf-61b8-41eb-9476-92479d9209d2\0d
...
remove:C:\Users\Default\AppData\Local\Temp\spark-371eb96b-738f-4f6f-97c9-7a694afc34cc\blockmgr-0691abbf-61b8-41eb-9476-92479d9209d2
remove:C:\Users\Default\AppData\Local\Temp\spark-371eb96b-738f-4f6f-97c9-7a694afc34cc
remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\spark-c0db2905-02f1-4bbb-811b-3cf910dee7cf\userFiles-76f038fe-4e25-4622-b8c1-74481ffb6eb9
```
_residual system directory is_
`C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\spark-c0db2905-02f1-4bbb-811b-3cf910dee7cf`
**After changes**
> Run the unit test: SortShuffleManager properly cleans up files for
shuffles that use the serialized path
```
remove:C:\Users\Default\AppData\Local\Temp\spark-c2a131f1-08a5-428c-bed4-e29cfcacbbb9
remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\spark-3a7708e1-56fb-40ed-890f-6a8cc80d7473\userFiles-68974f6c-d4fc-4f8a-b66f-01a25ed12c44
```
> Run the unit test: SortShuffleManager properly cleans up files for
shuffles that use the deserialized path
```
remove:C:\Users\Default\AppData\Local\Temp\spark-07ef44ee-c165-4637-a52d-daf2f8780a50
remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\spark-3a7708e1-56fb-40ed-890f-6a8cc80d7473\userFiles-19e1cd23-077d-4d05-a676-6d3f8f2cfa39
remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\spark-3a7708e1-56fb-40ed-890f-6a8cc80d7473
remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\blockmgr-73e600e1-0ff7-4d79-8da0-ffac572be0d3\0c
...
remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\blockmgr-73e600e1-0ff7-4d79-8da0-ffac572be0d3
remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\blockmgr-815208b6-c080-4707-8553-a5cec9709e1e\0c
...
remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\blockmgr-815208b6-c080-4707-8553-a5cec9709e1e
remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b
```
So, after changes, remove file sequence has also changed.
thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
