[jira] [Commented] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[ https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211722#comment-17211722 ] Hadoop QA commented on HADOOP-16878: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 33m 7s{color} | | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 54s{color} | | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m 53s{color} | | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 25m 43s{color} | | {color:green} trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 49s{color} | | {color:green} trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 49s{color} | | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 52s{color} | | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 21m 41s{color} | | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s{color} | | {color:green} trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 0s{color} | | {color:green} trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m 20s{color} | | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 32s{color} | | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 3s{color} | | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 36s{color} | | {color:green} the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m 36s{color} | | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 48s{color} | | {color:green} the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m 48s{color} | | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m 0s{color} | | {color:green} The patch has no blanks issues. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 45s{color} | | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 53s{color} | | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 24m 24s{color} | | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s{color} | | {color:green} the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 53s{color} | | {color:green} the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 23s{color} |
[jira] [Commented] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[ https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071438#comment-17071438 ] Ayush Saxena commented on HADOOP-16878: --- Seems the Jenkins got triggered m and the test failed there too: https://builds.apache.org/job/PreCommit-HADOOP-Build/16805/testReport/org.apache.hadoop.hdfs/TestDistributedFileSystem/testCopyBetweenFsEqualPath/ Give a check > Copy command in FileUtil to throw an exception if the source and destination > is the same > > > Key: HADOOP-16878 > URL: https://issues.apache.org/jira/browse/HADOOP-16878 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Attachments: hdfsTest.patch > > > We encountered an error during a test in our QE when the file destination and > source path were the same. This happened during an ADLS test, and there were > no meaningful error messages, so it was hard to find the root cause of the > failure. > The error we saw was that file size has changed during the copy operation. > The new file creation in the destination - which is the same as the source - > creates a file and sets the file length to zero. After this, getting the > source file will fail because the sile size changed during the operation. > I propose a solution to at least log in error level in the {{FileUtil}} if > the source and destination of the copy operation is the same, so debugging > issues like this will be easier in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[ https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071240#comment-17071240 ] Hadoop QA commented on HADOOP-16878: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 26s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 2s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 20m 49s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 51s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 32s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 30s{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 51s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 11s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}155m 34s{color} | {color:red} hadoop-hdfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 6s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}296m 0s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-common-project/hadoop-common | | | Nullcheck of src at line 400 of value previously dereferenced in org.apache.hadoop.fs.FileUtil.copy(FileSystem, FileStatus, FileSystem, Path, boolean, boolean, Configuration) At FileUtil.java:400 of value previously dereferenced in org.apache.hadoop.fs.FileUtil.copy(FileSystem, FileStatus, FileSystem, Path, boolean, boolean, Configuration) At FileUtil.java:[line 400] | | Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer | | | hadoop.hdfs.TestDistributedFileSystem | | | hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage | | | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery | | | hadoop.hdfs.server.namenode.TestFsck | | |
[jira] [Commented] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[ https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071107#comment-17071107 ] Ayush Saxena commented on HADOOP-16878: --- The Current PR still, fails for me : {code:java} [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Running org.apache.hadoop.hdfs.TestDistributedFileSystem [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.629 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestDistributedFileSystem [ERROR] testCopyBetweenFsEqualPath(org.apache.hadoop.hdfs.TestDistributedFileSystem) Time elapsed: 8.346 s <<< FAILURE! java.lang.AssertionError: Expected a org.apache.hadoop.fs.PathOperationException to be thrown, but got the result: : true at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384) at org.apache.hadoop.hdfs.TestDistributedFileSystem.testCopyBetweenFsEqualPath(TestDistributedFileSystem.java:2105) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) [INFO] [INFO] Results: [INFO] [ERROR] Failures: [ERROR] TestDistributedFileSystem.testCopyBetweenFsEqualPath:2105 Expected a org.apache.hadoop.fs.PathOperationException to be thrown, but got the result: : true [INFO] [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0 [INFO] {code} Can you apply the patch and run {{mvn test -Dtest=TestDistributedFileSystem#testCopyBetweenFsEqualPath}} > Copy command in FileUtil to throw an exception if the source and destination > is the same > > > Key: HADOOP-16878 > URL: https://issues.apache.org/jira/browse/HADOOP-16878 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Attachments: hdfsTest.patch > > > We encountered an error during a test in our QE when the file destination and > source path were the same. This happened during an ADLS test, and there were > no meaningful error messages, so it was hard to find the root cause of the > failure. > The error we saw was that file size has changed during the copy operation. > The new file creation in the destination - which is the same as the source - > creates a file and sets the file length to zero. After this, getting the > source file will fail because the sile size changed during the operation. > I propose a solution to at least log in error level in the {{FileUtil}} if > the source and destination of the copy
[jira] [Commented] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[ https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071074#comment-17071074 ] Gabor Bota commented on HADOOP-16878: - Yeah, I already updated the PR with that, but I cannot see this failing, which is strange. > Copy command in FileUtil to throw an exception if the source and destination > is the same > > > Key: HADOOP-16878 > URL: https://issues.apache.org/jira/browse/HADOOP-16878 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Attachments: hdfsTest.patch > > > We encountered an error during a test in our QE when the file destination and > source path were the same. This happened during an ADLS test, and there were > no meaningful error messages, so it was hard to find the root cause of the > failure. > The error we saw was that file size has changed during the copy operation. > The new file creation in the destination - which is the same as the source - > creates a file and sets the file length to zero. After this, getting the > source file will fail because the sile size changed during the operation. > I propose a solution to at least log in error level in the {{FileUtil}} if > the source and destination of the copy operation is the same, so debugging > issues like this will be easier in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[ https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071048#comment-17071048 ] Ayush Saxena commented on HADOOP-16878: --- {code:java} if (dst != null && src != null && srcFS.makeQualified(src).equals(dstFS.makeQualified(dst))) { {code} Changing the if condition like this fixes it for me. > Copy command in FileUtil to throw an exception if the source and destination > is the same > > > Key: HADOOP-16878 > URL: https://issues.apache.org/jira/browse/HADOOP-16878 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Attachments: hdfsTest.patch > > > We encountered an error during a test in our QE when the file destination and > source path were the same. This happened during an ADLS test, and there were > no meaningful error messages, so it was hard to find the root cause of the > failure. > The error we saw was that file size has changed during the copy operation. > The new file creation in the destination - which is the same as the source - > creates a file and sets the file length to zero. After this, getting the > source file will fail because the sile size changed during the operation. > I propose a solution to at least log in error level in the {{FileUtil}} if > the source and destination of the copy operation is the same, so debugging > issues like this will be easier in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[ https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071045#comment-17071045 ] Ayush Saxena commented on HADOOP-16878: --- Thanx [~gabor.bota] for checking, I added my test with your code, Can you help check once more. I too will try to figure out, if I am not missing anything. > Copy command in FileUtil to throw an exception if the source and destination > is the same > > > Key: HADOOP-16878 > URL: https://issues.apache.org/jira/browse/HADOOP-16878 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Attachments: hdfsTest.patch > > > We encountered an error during a test in our QE when the file destination and > source path were the same. This happened during an ADLS test, and there were > no meaningful error messages, so it was hard to find the root cause of the > failure. > The error we saw was that file size has changed during the copy operation. > The new file creation in the destination - which is the same as the source - > creates a file and sets the file length to zero. After this, getting the > source file will fail because the sile size changed during the operation. > I propose a solution to at least log in error level in the {{FileUtil}} if > the source and destination of the copy operation is the same, so debugging > issues like this will be easier in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[ https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071042#comment-17071042 ] Ayush Saxena commented on HADOOP-16878: --- Thanx [~gabor.bota] for checking, I attached the patch with the test, Can you check, if that is able to repro for you? > Copy command in FileUtil to throw an exception if the source and destination > is the same > > > Key: HADOOP-16878 > URL: https://issues.apache.org/jira/browse/HADOOP-16878 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Attachments: hdfsTest.patch > > > We encountered an error during a test in our QE when the file destination and > source path were the same. This happened during an ADLS test, and there were > no meaningful error messages, so it was hard to find the root cause of the > failure. > The error we saw was that file size has changed during the copy operation. > The new file creation in the destination - which is the same as the source - > creates a file and sets the file length to zero. After this, getting the > source file will fail because the sile size changed during the operation. > I propose a solution to at least log in error level in the {{FileUtil}} if > the source and destination of the copy operation is the same, so debugging > issues like this will be easier in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[ https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071038#comment-17071038 ] Gabor Bota commented on HADOOP-16878: - [~ayushtkn], I was not able to reproduce this test failure. How did you run the test? I added a check to be sure that {{dst}} and {{src}} are not null before calling .toUri(), but I'd be interested in how to reproduce the issue with {{testCopyBetweenFsEqualPath}} . > Copy command in FileUtil to throw an exception if the source and destination > is the same > > > Key: HADOOP-16878 > URL: https://issues.apache.org/jira/browse/HADOOP-16878 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > We encountered an error during a test in our QE when the file destination and > source path were the same. This happened during an ADLS test, and there were > no meaningful error messages, so it was hard to find the root cause of the > failure. > The error we saw was that file size has changed during the copy operation. > The new file creation in the destination - which is the same as the source - > creates a file and sets the file length to zero. After this, getting the > source file will fail because the sile size changed during the operation. > I propose a solution to at least log in error level in the {{FileUtil}} if > the source and destination of the copy operation is the same, so debugging > issues like this will be easier in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[ https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069312#comment-17069312 ] Ayush Saxena commented on HADOOP-16878: --- Thanx [~gabor.bota] for the report and PR. I tried this with a quick UT in HDFS. Seems I didn't get the exception thrown : Sharing the UT : {code:java} @Test public void testCopyBetweenFsEqualPath() throws Exception { Configuration conf = getTestConfiguration(); try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).build()) { cluster.waitActive(); final DistributedFileSystem dfs = cluster.getFileSystem(); Path filePath = new Path("/dir/file"); dfs.create(filePath).close(); FileStatus fstatus = dfs.getFileStatus(filePath); LambdaTestUtils.intercept(PathOperationException.class, () -> FileUtil.copy(dfs, fstatus, dfs, filePath, false, true, conf)); } } {code} (Can place in this TestDistributedFileSystem.java) The reason being : {code:java} if (src.toUri().equals(dst.toUri())) { {code} In {{src}} the scheme is present but in {{dst}} it isn't present. Can you give a check once. > Copy command in FileUtil to throw an exception if the source and destination > is the same > > > Key: HADOOP-16878 > URL: https://issues.apache.org/jira/browse/HADOOP-16878 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > We encountered an error during a test in our QE when the file destination and > source path were the same. This happened during an ADLS test, and there were > no meaningful error messages, so it was hard to find the root cause of the > failure. > The error we saw was that file size has changed during the copy operation. > The new file creation in the destination - which is the same as the source - > creates a file and sets the file length to zero. After this, getting the > source file will fail because the sile size changed during the operation. > I propose a solution to at least log in error level in the {{FileUtil}} if > the source and destination of the copy operation is the same, so debugging > issues like this will be easier in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16878) Copy command in FileUtil to throw an exception if the source and destination is the same
[ https://issues.apache.org/jira/browse/HADOOP-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066757#comment-17066757 ] Gabor Bota commented on HADOOP-16878: - PR is up with the small change and test. > Copy command in FileUtil to throw an exception if the source and destination > is the same > > > Key: HADOOP-16878 > URL: https://issues.apache.org/jira/browse/HADOOP-16878 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > We encountered an error during a test in our QE when the file destination and > source path were the same. This happened during an ADLS test, and there were > no meaningful error messages, so it was hard to find the root cause of the > failure. > The error we saw was that file size has changed during the copy operation. > The new file creation in the destination - which is the same as the source - > creates a file and sets the file length to zero. After this, getting the > source file will fail because the sile size changed during the operation. > I propose a solution to at least log in error level in the {{FileUtil}} if > the source and destination of the copy operation is the same, so debugging > issues like this will be easier in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org