[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3

2020-04-07 Thread GitBox 
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls 
getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-610313950
 
 
   thx


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3

2020-04-06 Thread GitBox 
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls 
getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-609848048
 
 
   line length.
   ```
   
./hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java:231:
String text = String.format("Source %s; dest %s: wrong ", srcStatus, 
destStatus);: Line is longer than 80 characters (found 85). [LineLength]
   
./hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java:232:
// FileStatus.equals only compares path field, must explicitly compare all 
fields: Line is longer than 80 characters (found 85). [LineLeng
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3

2020-04-06 Thread GitBox 
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls 
getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-609764508
 
 
   local test run happy; the test failures are close but probably unrelated.
   ```
   [INFO] Running org.apache.hadoop.tools.TestDistCpSystem
   [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
54.917 s - in org.apache.hadoop.tools.TestDistCpSystem
   [INFO] 
   [INFO] Results:
   [INFO] 
   [INFO] Tests run: 347, Failures: 0, Errors: 0, Skipped: 0
   [INFO] 
   ```
   
   
   One thing to worry about: yetus doesn't normally run distCP tests because 
there are so few patches against it. Could be an intrinsic problem.
   
   I'm going to rebase and resubmitted to see what happens.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3

2020-04-06 Thread GitBox 
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls 
getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-609722501
 
 
   error is minidfs cluster OOM


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3

2020-04-06 Thread GitBox 
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls 
getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-609717556
 
 
   you can set it in an API; I think it's propagating it so that even if you 
are doing an incremental write/append/update the replication factor will be 
updated
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3

2020-04-03 Thread GitBox 
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls 
getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-608343496
 
 
   Various tests in .TestDistCpUtils failing, tests whose assertion's dont 
include error text. @mukund-thakur  -this is a classic example of why I insist 
all new asserts are meaningful :(


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3

2020-04-03 Thread GitBox 
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls 
getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-608336023
 
 
   yeah, looks like I didn't test for regressions. Sill, yetus gets to do that 
for github PRs


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3

2020-04-02 Thread GitBox 
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls 
getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-607972418
 
 
   adds a quick fix and tests; verified that without the changes in production 
code the test fails
   
   ```
   
   java.io.FileNotFoundException: File does not exist: 
/tmp/testSkipsNeedlessAttributes/dest
   
at 
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1729)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1722)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1737)
at 
org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:210)
at 
org.apache.hadoop.tools.util.TestDistCpUtils.testSkipsNeedlessAttributes(TestDistCpUtils.java:234)
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org