[
https://issues.apache.org/jira/browse/HDFS-13509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16457061#comment-16457061
]
Xiao Liang commented on HDFS-13509:
-----------------------------------
And my local test result for trunk without [^HDFS-13509.000.patch] is:
{color:#d04437}[INFO]
-------------------------------------------------------{color}
{color:#d04437}[INFO] T E S T S{color}
{color:#d04437}[INFO]
-------------------------------------------------------{color}
{color:#d04437}[INFO] Running org.apache.hadoop.hdfs.TestFileAppend{color}
{color:#d04437}[ERROR] Tests run: 13, Failures: 0, Errors: 1, Skipped: 0, Time
elapsed: 59.103 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestFileAppend{color}
{color:#d04437}[ERROR]
testBreakHardlinksIfNeeded(org.apache.hadoop.hdfs.TestFileAppend) Time elapsed:
7.725 s <<< ERROR!{color}
{color:#d04437}java.io.IOException: Unable to rename
D:\Git\Hadoop\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\data\data1\current\BP-991234096-10.123.154.103-1524862692241\current\finalized\subdir0\subdir0\blk_1073741825.unlinked
to
D:\Git\Hadoop\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\data\data1\current\BP-991234096-10.123.154.103-1524862692241\current\finalized\subdir0\subdir0\blk_1073741825{color}
{color:#d04437} at
org.apache.hadoop.fs.FileUtil.replaceFile(FileUtil.java:1369){color}
{color:#d04437} at
org.apache.hadoop.hdfs.server.datanode.FileIoProvider.replaceFile(FileIoProvider.java:533){color}
{color:#d04437} at
org.apache.hadoop.hdfs.server.datanode.LocalReplica.breakHardlinks(LocalReplica.java:200){color}
{color:#d04437} at
org.apache.hadoop.hdfs.server.datanode.LocalReplica.breakHardLinksIfNeeded(LocalReplica.java:239){color}
{color:#d04437} at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetTestUtil.breakHardlinksIfNeeded(FsDatasetTestUtil.java:71){color}
{color:#d04437} at
org.apache.hadoop.hdfs.TestFileAppend.testBreakHardlinksIfNeeded(TestFileAppend.java:162){color}
{color:#d04437} at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method){color}
{color:#d04437} at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62){color}
{color:#d04437} at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){color}
{color:#d04437} at java.lang.reflect.Method.invoke(Method.java:498){color}
{color:#d04437} at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47){color}
{color:#d04437} at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12){color}
{color:#d04437} at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44){color}
{color:#d04437} at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17){color}
{color:#d04437} at
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271){color}
{color:#d04437} at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70){color}
{color:#d04437} at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50){color}
{color:#d04437} at
org.junit.runners.ParentRunner$3.run(ParentRunner.java:238){color}
{color:#d04437} at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63){color}
{color:#d04437} at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236){color}
{color:#d04437} at
org.junit.runners.ParentRunner.access$000(ParentRunner.java:53){color}
{color:#d04437} at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229){color}
{color:#d04437} at
org.junit.runners.ParentRunner.run(ParentRunner.java:309){color}
{color:#d04437} at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365){color}
{color:#d04437} at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273){color}
{color:#d04437} at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238){color}
{color:#d04437} at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159){color}
{color:#d04437} at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379){color}
{color:#d04437} at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340){color}
{color:#d04437} at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125){color}
{color:#d04437} at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413){color}
{color:#d04437}[INFO]{color}
{color:#d04437}[INFO] Results:{color}
{color:#d04437}[INFO]{color}
{color:#d04437}[ERROR] Errors:{color}
{color:#d04437}[ERROR] TestFileAppend.testBreakHardlinksIfNeeded:162 » IO
Unable to rename D:\Git\Had...{color}
{color:#d04437}[INFO]{color}
{color:#d04437}[ERROR] Tests run: 13, Failures: 0, Errors: 1, Skipped: 0{color}
{color:#d04437}[INFO]{color}
{color:#d04437}[INFO]
------------------------------------------------------------------------{color}
{color:#d04437}[INFO] BUILD FAILURE{color}
{color:#d04437}[INFO]
------------------------------------------------------------------------{color}
With [^HDFS-13509.000.patch] it's:
{color:#14892c}[INFO]
-------------------------------------------------------{color}
{color:#14892c}[INFO] T E S T S{color}
{color:#14892c}[INFO]
-------------------------------------------------------{color}
{color:#14892c}[INFO] Running org.apache.hadoop.hdfs.TestFileAppend{color}
{color:#14892c}[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 53.868 s - in org.apache.hadoop.hdfs.TestFileAppend{color}
{color:#14892c}[INFO]{color}
{color:#14892c}[INFO] Results:{color}
{color:#14892c}[INFO]{color}
{color:#14892c}[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0{color}
{color:#14892c}[INFO]{color}
{color:#14892c}[INFO]{color}
{color:#14892c}[INFO] --- maven-antrun-plugin:1.7:run (hdfs-test-bats-driver) @
hadoop-hdfs ---{color}
{color:#14892c}[INFO] Executing tasks{color}
{color:#14892c}main:{color}
{color:#14892c} [exec] which: no bats in
(/mingw64/bin:/usr/bin:/c/Users/helian/bin:/d/Git/MT/Tools/java/bin:/d/Git/MT/Tools/java/jre/bin/server:/d/Git/MT/Tools/Maven/bin:/d/Git/MT/Tools/protobuf/bin:/d/Git/MT/Tools/Cmake/bin:/d/Git/MT/Tools/zlib128:/d/Git/MT/Tools/Python34:/bin/git.exe:/d/Git/MT/Tools/Python34/Scripts:/c/Windows/Microsoft.NET/Framework64/v4.0.30319:/bin:/c/Program
Files/Microsoft MPI/Bin:/c/ProgramData/Oracle/Java/javapath:/c/Program Files
(x86)/Intel/iCLS Client:/c/Program Files/Intel/iCLS Client:/c/Program
Files/Common Files/Microsoft Shared/Microsoft Online Services:/c/Program Files
(x86)/Common Files/Microsoft Shared/Microsoft Online
Services:/c/windows/system32:/c/windows:/c/windows/System32/Wbem:/c/windows/System32/WindowsPowerShell/v1.0:/c/Program
Files/Intel/Intel(R) Management Engine Components/DAL:/c/Program
Files/Intel/Intel(R) Management Engine Components/IPT:/c/Program Files
(x86)/Intel/Intel(R) Management Engine Components/DAL:/c/Program Files
(x86)/Intel/Intel(R) Management Engine Components/IPT:/c/Program Files
(x86)/ATI Technologies/ATI.ACE/Core-Static:/c/Users/helian/.dnx/bin:/c/Program
Files/Microsoft DNX/Dnvm:/c/Program Files/Microsoft SQL
Server/120/Tools/Binn:/c/Program Files (x86)/nodejs:/c/Program
Files/Microsoft/Web Platform Installer:/c/Program Files (x86)/Magellan Toolset
4.75:/tmp/pipelinetools:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/System32/Wbem:/c/WINDOWS/System32/WindowsPowerShell/v1.0:/c/Program
Files (x86)/Skype/Phone:/c/Program Files/Microsoft SQL
Server/130/Tools/Binn:/c/GITCF:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/System32/Wbem:/c/WINDOWS/System32/WindowsPowerShell/v1.0:/c/Program
Files (x86)/Microsoft SQL Server/100/Tools/Binn:/c/Program Files/Microsoft SQL
Server/100/Tools/Binn:/c/Program Files/Microsoft SQL
Server/100/DTS/Binn:/c/Program Files (x86)/scala/bin:/c/Program
Files/WIDCOMM/Bluetooth Software:/c/Program Files/WIDCOMM/Bluetooth
Software/syswow64:/c/Program Files/dotnet:/c/Program Files
(x86)/GtkSharp/2.12/bin:/cmd:/c/Program Files
(x86)/Gow/bin:/c/Users/helian/.dnx/bin:/c/Users/helian/AppData/Roaming/npm:/e/ClientGoldEnlistment:/c/Program
Files (x86)/Microsoft VS
Code/bin:/c/Users/helian/AppData/Local/Microsoft/WindowsApps:/c/Program Files
(x86)/Microsoft Visual Studio 14.0/Common7/IDE){color}
{color:#14892c} [exec]{color}
{color:#14892c} [exec]{color}
{color:#14892c} [exec] ERROR: bats not installed. Skipping bash tests.{color}
{color:#14892c} [exec] ERROR: Please install bats as soon as possible.{color}
{color:#14892c} [exec]{color}
{color:#14892c} [exec]{color}
{color:#14892c}[INFO] Executed tasks{color}
{color:#14892c}[INFO]
------------------------------------------------------------------------{color}
{color:#14892c}[INFO] BUILD SUCCESS{color}
{color:#14892c}[INFO]
------------------------------------------------------------------------{color}
{color:#14892c}[INFO] Total time: 01:03 min{color}
{color:#14892c}[INFO] Finished at: 2018-04-27T11:55:18-07:00{color}
{color:#14892c}[INFO] Final Memory: 35M/686M{color}
{color:#14892c}[INFO]
------------------------------------------------------------------------{color}
The tests are passed, I also verified them in my IDE, the bats error should not
be related.
> Bug fix for breakHardlinks() of ReplicaInfo/LocalReplica, and fix
> TestFileAppend failures on Windows
> ----------------------------------------------------------------------------------------------------
>
> Key: HDFS-13509
> URL: https://issues.apache.org/jira/browse/HDFS-13509
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Xiao Liang
> Assignee: Xiao Liang
> Priority: Major
> Labels: windows
> Attachments: HDFS-13509-branch-2.000.patch, HDFS-13509.000.patch
>
>
> breakHardlinks() of ReplicaInfo(branch-2)/LocalReplica(trunk) replaces file
> while the source is still opened as input stream, which will fail and throw
> exception on Windows. It's the cause of unit test case
> org.apache.hadoop.hdfs.TestFileAppend#testBreakHardlinksIfNeeded failure on
> Windows.
> Other test cases of TestFileAppend fail randomly on Windows due to sharing
> the same test folder, and the solution is using randomized base dir of
> MiniDFSCluster via HDFS-13408
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
