[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-12-15 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-745459150


   Closing as nobody was reviewing this and its impossible to maintain the 
interface and impl branches. The combined -common and -aws changes make for a 
bigger pr, but each patch has gone in separately, It will be possible to merge 
the changes into Hadoop as two separate patches 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-10-29 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-718839259


   This PR won't merge right now due to the move to shaded Preconditons. The PR 
#2324 has been rebased; I'll be rebuilding this PR from that one as a trunk + 
one big fat patch



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-10-27 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-717435068


   Also, should use java.io.UncheckedIOException for the wrapper class for an 
IOE. That's new in Java8, but as its public API there, what we should adopt. 
This is good, even if its a bit more work



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-10-13 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-707756428


   build failure was in native IO. full PR in #2324 didn't show this. I'd 
rebase this PR except it would complicate everything else too much



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-10-09 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-705609349


   javac failures on newly deprecated static fs.impl.FutureIOSupport methods 
I'm retaining to avoiding breaking things; checkstyle on 4 javadoc lines which 
are too wide because they reference other classes
   
   ```
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:41:
   * @deprecated use {@link 
org.apache.hadoop.util.functional.FunctionRaisingIOE}: Line is longer than 80 
characters (found 81). [LineLength]
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:54:
   * @deprecated use {@link 
org.apache.hadoop.util.functional.BiFunctionRaisingIOE}: Line is longer than 80 
characters (found 83). [LineLength]
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:65:
   * @deprecated use {@link 
org.apache.hadoop.util.functional.CallableRaisingIOE}: Line is longer than 80 
characters (found 81). [LineLength]
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/package-info.java:28:
 * {@link org.apache.hadoop.fs.statistics.IOStatisticsSource#getIOStatistics()} 
.: Line is longer than 80 characters (found 81). [LineLength]
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-10-08 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-705609349


   javac failures on newly deprecated static fs.impl.FutureIOSupport methods 
I'm retaining to avoiding breaking things; checkstyle on 4 javadoc lines which 
are too wide because they reference other classes
   
   ```
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:41:
   * @deprecated use {@link 
org.apache.hadoop.util.functional.FunctionRaisingIOE}: Line is longer than 80 
characters (found 81). [LineLength]
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:54:
   * @deprecated use {@link 
org.apache.hadoop.util.functional.BiFunctionRaisingIOE}: Line is longer than 80 
characters (found 83). [LineLength]
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:65:
   * @deprecated use {@link 
org.apache.hadoop.util.functional.CallableRaisingIOE}: Line is longer than 80 
characters (found 81). [LineLength]
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/package-info.java:28:
 * {@link org.apache.hadoop.fs.statistics.IOStatisticsSource#getIOStatistics()} 
.: Line is longer than 80 characters (found 81). [LineLength]
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-10-02 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-702777116


   Yetus: javac is all about the deprecation changes fixed in #2324 
   
   checkstyles from new tests & code
   ```
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:41:
   * @deprecated use {@link 
org.apache.hadoop.util.functional.FunctionRaisingIOE}: Line is longer than 80 
characters (found 81). [LineLength]
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:54:
   * @deprecated use {@link 
org.apache.hadoop.util.functional.BiFunctionRaisingIOE}: Line is longer than 80 
characters (found 83). [LineLength]
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:65:
   * @deprecated use {@link 
org.apache.hadoop.util.functional.CallableRaisingIOE}: Line is longer than 80 
characters (found 81). [LineLength]
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/BufferedIOStatisticsInputStream.java:51:
  /**: First sentence should end with a period. [JavadocStyle]
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/package-info.java:28:
 * {@link org.apache.hadoop.fs.statistics.IOStatisticsSource#getIOStatistics()} 
.: Line is longer than 80 characters (found 81). [LineLength]
   
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/RemoteIterators.java:412:
/**: First sentence should end with a period. [JavadocStyle]
   
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/statistics/TestDurationTracking.java:112:
(Integer x) -> x);: 'lambda arguments' has incorrect indentation level 
8, expected level should be one of the following: 10, 12. [Indentation]
   
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/statistics/TestDurationTracking.java:200:
  /**: First sentence should end with a period. [JavadocStyle]
   ```
   will fix those



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-10-01 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-702364693


   @mehakmeet 
   * duration tracking for classic function. Issue: are the names of the 
`trackDuration` calls correct now?
   * class DurationStatisticSummary to store and extract duration stats from a 
statistic.
   * which is used in the tests to verify that the new functions all work 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-09-30 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-701491715


   > error. This is solved when I add ".mean" at the end of the DurationTracker 
stat name. The same with Max(".max" has to be added) and Min(".min" has to be 
added) to fetch their values.
   > Was thinking if we could add it to the getter rather than having to append 
with stat name.
   
   I'm a bit reluctant to, as then people would ask about failures next. What 
could be handy would be something in the support class to get all stats for a 
duration (or null), with some struct to contain them all. e.g
   
   `fetchDurationStatistics(IOStatistics, key) -> {count, min, max ,mean, 
failed, failed.min, failed.max, failed.mean)`, and you'd then work off that. 
Seem good? 
   Troublespot: what if only some of the values were found? they'd be null in 
the result. Maybe we'd let you ask for the success/failure stats separately, 
deal with it that way, and have some `boolean isComplete()` probe to check all 
are set



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-09-29 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-700777072


   Update: you will get the ability to wrap a Callable, CallableRaisingIOE or 
functionRaisingIOE with a duration tracking, which then returns a new instance 
of the same
   
   ```java
   submit(unboundedThreadPool,
 trackDurationOfCallable(
 listingContext.getDurationTrackerFactory(),
 OBJECT_LIST_REQUEST,
 () -> listObjects(request)));
   ```
   
   When there's a failure we update the failure count, move timings into the 
failures section. Attempts are counted for both, so success == attempts - 
failures.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-09-28 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-700065247


   > For DurationTrackers in IOStatisticsStore() if we add a tracker in a try 
block, what happens to it in case of failure should be looked at to avoid 
inaccurate values for the trackers.
   
   I was thinking about failure reporting myself
   
   - we may want to count failures
   - any failure which with a longer or shorter duration than successful 
operations Will skew the results. Example: network failures -> long durations; 
auth failures -> short ones.
   
   At the same time, try-with-resources is nice. What to do?
   
   For each set of duration stats, we add counter/mean/min/max of failures
   on a failure, those statistics are updated instead.
   
   Issue: how best to record a failure, given we can't get at the 
try-with-resources classes in catch or finally? I'd initially thought we could 
set it in the catch(), but it'd be out of scope.
   
   1. Pessimistic: assume that all attempts are failure, make last operation in 
every try clause set the success flag. Ugly.
   1. Move construction out of try-with-resources and instead explicit catch 
and finally. Differently ugly
   
   Fancy lambda-expression wrapper thing? Doable.
   
   ```
   object = DurationTrackerFactory.track("statistic", () ->
 s3.listObjects());
   ```
 
   Then in that code we'd put the code of option #2 in
   
   Fancy curried-function-Haskell-elitism option
   
   Duration tracker takes a function and returns a new one
   
   ```
   FunctionRaisingIOE track(String, FunctionRaisingIOE inner)
   ```
   
   you'd get a function back which you could then apply at leisure.
   
   ```
   DurationTrackerFactory.track("statistic", () ->
 s3.listObjects()).apply();
   ```
 
   Maybe worth doing both. I could also look at adding into the S3A Invoke 
code, as every iteration of a retried operation we'd want the statistic 
updated. 
   
   ```
   
   At the same time: this gets complex fast. Could we make the design of this a 
followup?
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-09-25 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-698303509


   findbugs
   ```
   module:hadoop-common-project/hadoop-common
   --
     | org.apache.hadoop.fs.statistics.IOStatisticsSnapshot's readObject method 
is synchronized At IOStatisticsSnapshot.java: At 
IOStatisticsSnapshot.java:[lines 241-254]
   ```
   
   The synchronized was lifted from TreeMap writeObject. When I turn it off 
I'll be reasonably confident that findbugs will start complaining about 
unsynchronized accesses to field. Let's see. If it blows up then I'll just 
disable the check here
   
   checkstyles are on the uses of the now deprecated methods; will move them on
   
   
   ```
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CallableSupplier.java:[34,32]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[82,32]
 [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[83,32]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[24,32]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CallableSupplier.java:[68,16]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:[436,51]
 [deprecation] FAST_UPLOAD in Constants has been deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[829,13]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1637,6]
 [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1637,25]
 [deprecation] CallableRaisingIOE in FunctionsRaisingIOE has been deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[103,18]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[121,18]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/ITestS3SelectMRJob.java:[33,32]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[48,32]
 [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[49,32]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java:[52,32]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/ITestS3SelectMRJob.java:[212,22]
 [deprecation] WrappedIOException in

[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-09-24 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-698303509


   findbugs
   ```
   module:hadoop-common-project/hadoop-common
   --
     | org.apache.hadoop.fs.statistics.IOStatisticsSnapshot's readObject method 
is synchronized At IOStatisticsSnapshot.java: At 
IOStatisticsSnapshot.java:[lines 241-254]
   ```
   
   The synchronized was lifted from TreeMap writeObject. When I turn it off 
I'll be reasonably confident that findbugs will start complaining about 
unsynchronized accesses to field. Let's see. If it blows up then I'll just 
disable the check here
   
   checkstyles are on the uses of the now deprecated methods; will move them on
   
   
   ```
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CallableSupplier.java:[34,32]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[82,32]
 [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[83,32]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[24,32]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CallableSupplier.java:[68,16]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:[436,51]
 [deprecation] FAST_UPLOAD in Constants has been deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[829,13]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1637,6]
 [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1637,25]
 [deprecation] CallableRaisingIOE in FunctionsRaisingIOE has been deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[103,18]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/RetryingCollection.java:[121,18]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/ITestS3SelectMRJob.java:[33,32]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[48,32]
 [deprecation] FunctionsRaisingIOE in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:[49,32]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java:[52,32]
 [deprecation] WrappedIOException in org.apache.hadoop.fs.impl has been 
deprecated
   [WARNING] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/ITestS3SelectMRJob.java:[212,22]
 [deprecation] WrappedIOException in

[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-09-22 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-696627308


   I see the problem. This PR only includes the common changes, but as that 
moved some fs.impl methods to being public in a new package, the uses in the s3 
code won't build. 
   ```
   [INFO] 

   [INFO] Total time:  19:15 min
   [INFO] Finished at: 2020-09-21T20:50:29Z
   [INFO] 

   [ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on 
project hadoop-aws: Compilation failure: Compilation failure: 
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[82,32]
 error: cannot find symbol
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1637,25]
 error: package FunctionsRaisingIOE does not exist
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1638,10]
 error: lambda expression not expected here
   [ERROR] -> [Help 1]
   [ERROR] 
   ```
   Briefly I will reinstate the delete class from fs.impl so everything builds 
again



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-09-22 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-696627308


   I see the problem. This PR only includes the common changes, but as that 
moved some fs.impl methods to being public in a new package, the uses in the s3 
code won't build. 
   ```
   [INFO] 

   [INFO] Total time:  19:15 min
   [INFO] Finished at: 2020-09-21T20:50:29Z
   [INFO] 

   [ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on 
project hadoop-aws: Compilation failure: Compilation failure: 
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[82,32]
 error: cannot find symbol
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1637,25]
 error: package FunctionsRaisingIOE does not exist
   [ERROR] 
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2323/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java:[1638,10]
 error: lambda expression not expected here
   [ERROR] -> [Help 1]
   [ERROR] 
   ```
   Briefly I will reinstate the delete class from fs.impl so everything builds 
again



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-09-21 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-696292801


   hadoop common part of #2069 : API and impl,
   
   While this is isolated for review, know that I'll be doing my changes on 
#2324 and cherry-picking here. It's just isolated for ease of review/to scare 
people less.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on pull request #2323: HADOOP-16830. Add public IOStatistics API.

2020-09-21 Thread GitBox 


steveloughran commented on pull request #2323:
URL: https://github.com/apache/hadoop/pull/2323#issuecomment-696292801


   hadoop common part of #2069 : API and impl,
   
   While this is isolated for review, know that I'll be doing my changes on 
#2324 and cherry-picking here. It's just isolated for ease of review/to scare 
people less.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org