[
https://issues.apache.org/jira/browse/FLINK-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098164#comment-17098164
]
Zhe Yu edited comment on FLINK-16198 at 5/3/20, 1:54 AM:
---------------------------------------------------------
h3. Failed test case 1: FileUtilsTest.testCompressionOnRelativePath
*Root cause*: failed to create directory using relative path when preparing
file for testing.
*Solution*: Since goal of this test case is to verify correctness of
FileUtils.compressDirectory and FileUtils.expandDirectory
Thus refactor verifyDirectoryCompression to prepare test data using absolute
path while we will still use relative path to test FileUtils method will be
sufficient.
h3. Failed test case 2: FileUtilsTest.testDeleteDirectoryConcurrently
*Root cause:* FileUtils.deleteDirectory is not thread safe. This test case is
not always failing. In my local env(MacOS), it failed 7,8 times out of 10. The
test case is creating 3 level file hierarchy like below:
a
- b
- e
- c
- d
Then the test case started 3 threads. All of them are trying to iterate from
root directory: a. Thus, race condition will happen. There will be a chance 2
threads trying to delete /a/b/e/ at the same time. When this happens the test
will file. Maybe I'm missing something given [this comment|#L248]
*Solution:* make [deleteFileOrDirectoryInternal|#L317] synchronized will
protect the test from failing and
[guardIfWindows|[https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/util/FileUtils.java#L389]]
can be removed. But it also means concurrent deletion is disabled if 2 threads
trying to delete from the same root.
If above proposal sounds reasonable to you, feel free to assign this issue to
me. I'm also more than glad to talk about both of them.
was (Author: fishzhe):
h3. Failed test case 1: FileUtilsTest.testCompressionOnRelativePath
*Root cause*: failed to create directory using relative path when preparing
file for testing.
*Solution*: Since goal of this test case is to verify correctness of
FileUtils.compressDirectory and FileUtils.expandDirectory
Thus refactor verifyDirectoryCompression to prepare test data using absolute
path while we will still use relative path to test FileUtils method will be
sufficient.
h3. Failed test case 2: FileUtilsTest.testDeleteDirectoryConcurrently
*Root cause:* FileUtils.deleteDirectory is not thread safe. This test case is
not always failing. In my local env(MacOS), it failed 7,8 times out of 10. The
test case is creating 3 level file hierarchy like below:
a
- b
- e
- c
- d
Then the test case started 3 threads. All of them are trying to iterate from
root directory: a. Thus, race condition will happen. There will be a chance 2
threads trying to delete /a/b/e/ at the same time. When this happens the test
will file. Maybe I'm missing something given [this comment|#L248]]
*Solution:* make [deleteFileOrDirectoryInternal|#L317]] synchronized will
protect the test from failing. But it also means concurrent deletion is
disabled if 2 threads trying to delete from the same root.
If above proposal sounds reasonable to you, feel free to assign this issue to
me. I'm also more than glad to talk about both of them.
> FileUtilsTest fails on Mac OS
> -----------------------------
>
> Key: FLINK-16198
> URL: https://issues.apache.org/jira/browse/FLINK-16198
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Tests
> Affects Versions: 1.11.0
> Reporter: Andrey Zagrebin
> Priority: Blocker
> Labels: starter
> Fix For: 1.11.0
>
>
> The following tests fail if run on Mac OS (IDE/maven).
>
> FileUtilsTest.testCompressionOnRelativePath:
> {code:java}
> java.nio.file.NoSuchFileException:
> ../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0c0000gn/T/junit6496651678375117676/compressDir/rootDirjava.nio.file.NoSuchFileException:
>
> ../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0c0000gn/T/junit6496651678375117676/compressDir/rootDir
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
> at java.nio.file.Files.createDirectory(Files.java:674) at
> org.apache.flink.util.FileUtilsTest.verifyDirectoryCompression(FileUtilsTest.java:440)
> at
> org.apache.flink.util.FileUtilsTest.testCompressionOnRelativePath(FileUtilsTest.java:261)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at
> org.junit.rules.RunRules.evaluate(RunRules.java:20) at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137) at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {code}
>
> FileUtilsTest.testDeleteDirectoryConcurrently:
> {code:java}
> java.nio.file.FileSystemException:
> /var/folders/67/v4yp_42d21j6_n8k1h556h0c0000gn/T/junit7558825557740784886/junit3566161583262218465/ab1fa0bde8b22cad58b717508c7a7300/121fdf5f7b057183843ed2e1298f9b66/6598025f390d3084d69c98b36e542fe2/8db7cd9c063396a19a86f5b63ce53f66:
> Invalid argument at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at
> sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
> at java.nio.file.Files.deleteIfExists(Files.java:1165)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:324)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at
> org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at
> org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:320)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at
> org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at
> org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:320)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at
> org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at
> org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:320)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at
> org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at
> org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:276)
> at
> org.apache.flink.util.FileUtilsTest$Deleter.go(FileUtilsTest.java:515)
> at
> org.apache.flink.core.testutils.CheckedThread.run(CheckedThread.java:74)java.nio.file.FileSystemException:
>
> /var/folders/67/v4yp_42d21j6_n8k1h556h0c0000gn/T/junit7558825557740784886/junit3566161583262218465/ab1fa0bde8b22cad58b717508c7a7300/121fdf5f7b057183843ed2e1298f9b66/6598025f390d3084d69c98b36e542fe2/8db7cd9c063396a19a86f5b63ce53f66:
> Invalid argument
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at
> sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
> at java.nio.file.Files.deleteIfExists(Files.java:1165)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:324)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at
> org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at
> org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:320)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at
> org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at
> org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:320)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at
> org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at
> org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:320)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at
> org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at
> org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:276)
> at
> org.apache.flink.util.FileUtilsTest$Deleter.go(FileUtilsTest.java:515)
> at
> org.apache.flink.core.testutils.CheckedThread.run(CheckedThread.java:74)%MCEPASTEBIN%{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
