[
https://issues.apache.org/jira/browse/HADOOP-18340?focusedWorklogId=796206&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796206
]
ASF GitHub Bot logged work on HADOOP-18340:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 28/Jul/22 23:26
Start Date: 28/Jul/22 23:26
Worklog Time Spent: 10m
Work Description: huaxiangsun commented on PR #4608:
URL: https://github.com/apache/hadoop/pull/4608#issuecomment-1198721463
> code in s3afs and the tests all look good.
>
> But i think the changes should be restricted to s3a fs, even if duplicates
a bit of the superclass.
>
> That whole section in the javadocs at the top of the Filesystem. class
explains why; you've already seen some of those test values but it's and
external implementations which we don't have control of. Adding anything is
making a commitment to preserve a new public API forever.
Thanks! Points well taken.
>
> I will be happier if you were just do it all in S3AFileSystem.
>
> override deleteOnExit(Path f) and
>
> 1. skip the exists check because it doesn't do the right thing if you call
the method while writing a file, because the file isn't visible until close.
saves HEAD/List probes too
> 2. save the list to a set local to s3a fs. you could make its getter
protected so that mock test can access it.
Got it, for s3a fs, the deleteOnExit set is going to be in s3a.
>
> now, what about an integration test too?
Will try to add an IT.
>
> add a test case which creates a new fs in the test case (keeping the
normal getFileSystem() fs for assertions)
>
> * adds a file which doesn't exist
> * adds a file which doesn't exist, then create it
> * adds a file which does exist
> * add a directory path
> * close the fs, use ContractTestUtils methods to assert the files and dirs
are gone.
>
> seem good?
Sounds good to me, thanks for the feedbacks. May come back to you for test
case, going to check it out first by myself.
Issue Time Tracking
-------------------
Worklog Id: (was: 796206)
Time Spent: 1h 40m (was: 1.5h)
> deleteOnExit does not work with S3AFileSystem
> ---------------------------------------------
>
> Key: HADOOP-18340
> URL: https://issues.apache.org/jira/browse/HADOOP-18340
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.3.3
> Reporter: Huaxiang Sun
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> When deleteOnExit is set on some paths, they are not removed when file system
> object is closed. The following exception is logged when printing out the
> exception in info log.
> {code:java}
> 2022-07-15 19:29:12,552 [main] INFO fs.FileSystem
> (FileSystem.java:processDeleteOnExit(1810)) - Ignoring failure to
> deleteOnExit for path /file, exception {}
> java.io.IOException: s3a://mock-bucket: FileSystem is closed!
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.checkNotClosed(S3AFileSystem.java:3887)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2333)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2355)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4402)
> at
> org.apache.hadoop.fs.FileSystem.processDeleteOnExit(FileSystem.java:1805)
> at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2669)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.close(S3AFileSystem.java:3830)
> at
> org.apache.hadoop.fs.s3a.TestS3AGetFileStatus.testFile(TestS3AGetFileStatus.java:87)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:258)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
