[jira] [Commented] (HADOOP-17010) Add queue capacity weights support in FairCallQueue

2020-04-28 Thread Fengnan Li (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095097#comment-17095097 ] Fengnan Li commented on HADOOP-17010: - Thanks so much [~csun]! > Add queue capacity weights

[GitHub] [hadoop] hadoop-yetus commented on pull request #1986: HADOOP-17019. Declare ProtobufHelper a public API

2020-04-28 Thread GitBox
hadoop-yetus commented on pull request #1986: URL: https://github.com/apache/hadoop/pull/1986#issuecomment-620984458 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m

[jira] [Updated] (HADOOP-17019) Declare ProtobufHelper a public API

2020-04-28 Thread Wei-Chiu Chuang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-17019: - Target Version/s: 3.3.1, 3.4.0 Opened a PR https://github.com/apache/hadoop/pull/1986

[GitHub] [hadoop] jojochuang opened a new pull request #1986: HADOOP-17019. Declare ProtobufHelper a public API

2020-04-28 Thread GitBox
jojochuang opened a new pull request #1986: URL: https://github.com/apache/hadoop/pull/1986 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[jira] [Updated] (HADOOP-17019) Declare ProtobufHelper a public API

2020-04-28 Thread Wei-Chiu Chuang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-17019: - Summary: Declare ProtobufHelper a public API (was: Declare Protobuf a public API) >

[jira] [Assigned] (HADOOP-17019) Declare ProtobufHelper a public API

2020-04-28 Thread Wei-Chiu Chuang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reassigned HADOOP-17019: Assignee: Wei-Chiu Chuang > Declare ProtobufHelper a public API >

[jira] [Created] (HADOOP-17019) Declare Protobuf a public API

2020-04-28 Thread Wei-Chiu Chuang (Jira)
Wei-Chiu Chuang created HADOOP-17019: Summary: Declare Protobuf a public API Key: HADOOP-17019 URL: https://issues.apache.org/jira/browse/HADOOP-17019 Project: Hadoop Common Issue Type:

[jira] [Updated] (HADOOP-16621) [pb-upgrade] Remove Protobuf classes from signatures of Public APIs

2020-04-28 Thread Wei-Chiu Chuang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-16621: - Hadoop Flags: Incompatible change,Reviewed (was: Reviewed) > [pb-upgrade] Remove

[jira] [Commented] (HADOOP-16621) [pb-upgrade] Remove Protobuf classes from signatures of Public APIs

2020-04-28 Thread Wei-Chiu Chuang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095002#comment-17095002 ] Wei-Chiu Chuang commented on HADOOP-16621: -- Updated RN to make Hadoop application developers

[jira] [Updated] (HADOOP-16621) [pb-upgrade] Remove Protobuf classes from signatures of Public APIs

2020-04-28 Thread Wei-Chiu Chuang (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-16621: - Release Note: Following APIs have been removed from Token.java to avoid protobuf

[jira] [Commented] (HADOOP-17010) Add queue capacity weights support in FairCallQueue

2020-04-28 Thread Hudson (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094926#comment-17094926 ] Hudson commented on HADOOP-17010: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18195 (See

[jira] [Updated] (HADOOP-17010) Add queue capacity weights support in FairCallQueue

2020-04-28 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17010: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Add queue capacity weights

[GitHub] [hadoop] sunchao commented on pull request #1977: HADOOP-17010. Add queue capacity support for FairCallQueue

2020-04-28 Thread GitBox
sunchao commented on pull request #1977: URL: https://github.com/apache/hadoop/pull/1977#issuecomment-620903073 Test failures do not look related. Merged. Thanks @fengnanli ! This is an automated message from the Apache Git

[GitHub] [hadoop] hadoop-yetus commented on pull request #1982: HADOOP-16830. IOStatistics API.

2020-04-28 Thread GitBox
hadoop-yetus commented on pull request #1982: URL: https://github.com/apache/hadoop/pull/1982#issuecomment-620889592 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m

[GitHub] [hadoop] hadoop-yetus commented on pull request #1980: HDFS-15301. statfs function in hdfs-fuse not working - fixed.

2020-04-28 Thread GitBox
hadoop-yetus commented on pull request #1980: URL: https://github.com/apache/hadoop/pull/1980#issuecomment-620740067 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m

[GitHub] [hadoop] mehakmeet commented on pull request #1918: HADOOP-16910: ABFS Streams to update FileSystem.Statistics counters on IO.

2020-04-28 Thread GitBox
mehakmeet commented on pull request #1918: URL: https://github.com/apache/hadoop/pull/1918#issuecomment-620737853 Ok, that is not the case with my configurations maybe. I am getting random fails and the test failed like 1/20 times for me. It doesn't break consecutively this much for me.

[GitHub] [hadoop] amarnathkarthik commented on pull request #1918: HADOOP-16910: ABFS Streams to update FileSystem.Statistics counters on IO.

2020-04-28 Thread GitBox
amarnathkarthik commented on pull request #1918: URL: https://github.com/apache/hadoop/pull/1918#issuecomment-620729423 Standalone and parallel fails with same error. Also i'd try to run at least multiple (10) times and failed every single time. `mvn -T 1C -Dparallel-tests=abfs

[GitHub] [hadoop] aryangupta1998 commented on pull request #1980: HDFS-15301. statfs function in hdfs-fuse not working - fixed.

2020-04-28 Thread GitBox
aryangupta1998 commented on pull request #1980: URL: https://github.com/apache/hadoop/pull/1980#issuecomment-620690131 > Changes looks good to me, especially based on the error message. Nice catch @aryangupta1998 an thank you for the PR. > > One thing I would like to be taken care

[GitHub] [hadoop] hadoop-yetus commented on pull request #1950: HADOOP-16586. ITestS3GuardFsck, others fails when run using a local m…

2020-04-28 Thread GitBox
hadoop-yetus commented on pull request #1950: URL: https://github.com/apache/hadoop/pull/1950#issuecomment-620626523 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m

[GitHub] [hadoop] mukund-thakur commented on pull request #1944: HADOOP-16900. Very large files can be truncated when written through S3AFileSystem

2020-04-28 Thread GitBox
mukund-thakur commented on pull request #1944: URL: https://github.com/apache/hadoop/pull/1944#issuecomment-620623345 Looks like the stackoverflow error is completely normal (as in process is just running out of stack memory) as it only happens when the number of parts is more that 4600.

[GitHub] [hadoop] mukund-thakur commented on pull request #1944: HADOOP-16900. Very large files can be truncated when written through S3AFileSystem

2020-04-28 Thread GitBox
mukund-thakur commented on pull request #1944: URL: https://github.com/apache/hadoop/pull/1944#issuecomment-620611177 Also the minimum upload size check is in the SDK side and it surfaces only when we issue finalizeMultipartUpload(). The reason we don't get size error during regular

[GitHub] [hadoop] mukund-thakur commented on pull request #1944: HADOOP-16900. Very large files can be truncated when written through S3AFileSystem

2020-04-28 Thread GitBox
mukund-thakur commented on pull request #1944: URL: https://github.com/apache/hadoop/pull/1944#issuecomment-620606478 > Also, I tried removing the 5 MB checks but the test fails with StackOverflow error because of the tail recursion present at Here is the stack trace. Also the

[GitHub] [hadoop] jasstionzyf opened a new pull request #1985: YARN-10248: when config allowed-gpu-devices , excluded GPUs still be visible to containers

2020-04-28 Thread GitBox
jasstionzyf opened a new pull request #1985: URL: https://github.com/apache/hadoop/pull/1985 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hadoop] jasstionzyf opened a new pull request #1984: when use allowed-gpu-devices, gpus can not exclude right

2020-04-28 Thread GitBox
jasstionzyf opened a new pull request #1984: URL: https://github.com/apache/hadoop/pull/1984 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number.

[GitHub] [hadoop] fapifta commented on pull request #1980: HDFS-15301. statfs function in hdfs-fuse not working - fixed.

2020-04-28 Thread GitBox
fapifta commented on pull request #1980: URL: https://github.com/apache/hadoop/pull/1980#issuecomment-620570881 Changes looks good to me, especially based on the error message. Nice catch @aryangupta1998 an thank you for the PR. One thing I would like to be taken care of is

[GitHub] [hadoop] iwasakims commented on pull request #1950: HADOOP-16586. ITestS3GuardFsck, others fails when run using a local m…

2020-04-28 Thread GitBox
iwasakims commented on pull request #1950: URL: https://github.com/apache/hadoop/pull/1950#issuecomment-620567567 I think ITestS3GuardToolLocal should always use LocalMetadataStore rather than skipping the tests based on the value of fs.s3a.metadatastore.impl.

[jira] [Comment Edited] (HADOOP-17018) Intermittent failing of ITestAbfsStreamStatistics in ABFS

2020-04-28 Thread Mehakmeet Singh (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094399#comment-17094399 ] Mehakmeet Singh edited comment on HADOOP-17018 at 4/28/20, 11:13 AM:

[jira] [Comment Edited] (HADOOP-17018) Intermittent failing of ITestAbfsStreamStatistics in ABFS

2020-04-28 Thread Mehakmeet Singh (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094399#comment-17094399 ] Mehakmeet Singh edited comment on HADOOP-17018 at 4/28/20, 11:11 AM:

[jira] [Commented] (HADOOP-17018) Intermittent failing of ITestAbfsStreamStatistics in ABFS

2020-04-28 Thread Mehakmeet Singh (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094399#comment-17094399 ] Mehakmeet Singh commented on HADOOP-17018: -- Stack trace   {code:java} 2020-04-28 15:49:04,106

[jira] [Created] (HADOOP-17018) Intermittent failing of ITestAbfsStreamStatistics in ABFS

2020-04-28 Thread Mehakmeet Singh (Jira)
Mehakmeet Singh created HADOOP-17018: Summary: Intermittent failing of ITestAbfsStreamStatistics in ABFS Key: HADOOP-17018 URL: https://issues.apache.org/jira/browse/HADOOP-17018 Project: Hadoop

[GitHub] [hadoop] mehakmeet commented on pull request #1918: HADOOP-16910: ABFS Streams to update FileSystem.Statistics counters on IO.

2020-04-28 Thread GitBox
mehakmeet commented on pull request #1918: URL: https://github.com/apache/hadoop/pull/1918#issuecomment-620529460 stack trace for the error: ``` 2020-04-28 15:49:04,106 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:execute(133)) - First execution of

[GitHub] [hadoop] steveloughran commented on pull request #1918: HADOOP-16910: ABFS Streams to update FileSystem.Statistics counters on IO.

2020-04-28 Thread GitBox
steveloughran commented on pull request #1918: URL: https://github.com/apache/hadoop/pull/1918#issuecomment-620528057 this sounds intermittent or configuration related This is an automated message from the Apache Git

[GitHub] [hadoop] steveloughran edited a comment on pull request #1918: HADOOP-16910: ABFS Streams to update FileSystem.Statistics counters on IO.

2020-04-28 Thread GitBox
steveloughran edited a comment on pull request #1918: URL: https://github.com/apache/hadoop/pull/1918#issuecomment-620528057 this sounds intermittent or configuration related. Does it show up in standalone runs or only in parallel ones?

[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #1918: HADOOP-16910: ABFS Streams to update FileSystem.Statistics counters on IO.

2020-04-28 Thread GitBox
hadoop-yetus removed a comment on pull request #1918: URL: https://github.com/apache/hadoop/pull/1918#issuecomment-604942333 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #1918: HADOOP-16910: ABFS Streams to update FileSystem.Statistics counters on IO.

2020-04-28 Thread GitBox
hadoop-yetus removed a comment on pull request #1918: URL: https://github.com/apache/hadoop/pull/1918#issuecomment-604965378 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec

[GitHub] [hadoop] steveloughran commented on pull request #1950: HADOOP-16586. ITestS3GuardFsck, others fails when run using a local m…

2020-04-28 Thread GitBox
steveloughran commented on pull request #1950: URL: https://github.com/apache/hadoop/pull/1950#issuecomment-620510583 hey, you are right, local s3guard runs do fail ``` [ERROR] testInitNegativeRead(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal) Time elapsed: 1.471 s

[jira] [Resolved] (HADOOP-16644) Retrive modtime of PUT file from store, via response or HEAD

2020-04-28 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16644. - Resolution: Won't Fix Update: put result doesn't include timestamp. unless we do a

[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #1747: HDFS-15042 Add more tests for ByteBufferPositionedReadable.

2020-04-28 Thread GitBox
hadoop-yetus removed a comment on pull request #1747: URL: https://github.com/apache/hadoop/pull/1747#issuecomment-570898301 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec

[GitHub] [hadoop] steveloughran commented on pull request #1627: HADOOP-16644. Do a HEAD after a PUT to get the modtime.

2020-04-28 Thread GitBox
steveloughran commented on pull request #1627: URL: https://github.com/apache/hadoop/pull/1627#issuecomment-620499528 Update: put result doesn't include timestamp. unless we do a HEAD every time, this is in trouble. I'm going to close the patch as is. If we were to revisit this it

[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #1950: HADOOP-16586. ITestS3GuardFsck, others fails when run using a local m…

2020-04-28 Thread GitBox
hadoop-yetus removed a comment on pull request #1950: URL: https://github.com/apache/hadoop/pull/1950#issuecomment-612266601 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec

[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #1950: HADOOP-16586. ITestS3GuardFsck, others fails when run using a local m…

2020-04-28 Thread GitBox
hadoop-yetus removed a comment on pull request #1950: URL: https://github.com/apache/hadoop/pull/1950#issuecomment-612092268 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec

[GitHub] [hadoop] steveloughran commented on a change in pull request #1950: HADOOP-16586. ITestS3GuardFsck, others fails when run using a local m…

2020-04-28 Thread GitBox
steveloughran commented on a change in pull request #1950: URL: https://github.com/apache/hadoop/pull/1950#discussion_r416469326 ## File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardToolLocal.java ## @@ -165,18 +165,19 @@ public

[GitHub] [hadoop] hadoop-yetus commented on pull request #1950: HADOOP-16586. ITestS3GuardFsck, others fails when run using a local m…

2020-04-28 Thread GitBox
hadoop-yetus commented on pull request #1950: URL: https://github.com/apache/hadoop/pull/1950#issuecomment-620478392 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m

[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1944: HADOOP-16900. Very large files can be truncated when written through S3AFileSystem

2020-04-28 Thread GitBox
mukund-thakur commented on a change in pull request #1944: URL: https://github.com/apache/hadoop/pull/1944#discussion_r416427061 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java ## @@ -394,6 +395,13 @@ public void close()

[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1944: HADOOP-16900. Very large files can be truncated when written through S3AFileSystem

2020-04-28 Thread GitBox
mukund-thakur commented on a change in pull request #1944: URL: https://github.com/apache/hadoop/pull/1944#discussion_r416419528 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/WriteOperationHelper.java ## @@ -400,18 +406,22 @@ public

[GitHub] [hadoop] mukund-thakur commented on pull request #1944: HADOOP-16900. Very large files can be truncated when written through S3AFileSystem

2020-04-28 Thread GitBox
mukund-thakur commented on pull request #1944: URL: https://github.com/apache/hadoop/pull/1944#issuecomment-620450785 I think it is kind of impossible to write integration test for this as we have 5 MB minimum block size check at multiple places. Even if we remove those checks, we will

[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1944: HADOOP-16900. Very large files can be truncated when written through S3AFileSystem

2020-04-28 Thread GitBox
mukund-thakur commented on a change in pull request #1944: URL: https://github.com/apache/hadoop/pull/1944#discussion_r416412072 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/InternalConstants.java ## @@ -93,4 +93,10 @@ private

[GitHub] [hadoop] mehakmeet commented on pull request #1918: HADOOP-16910: ABFS Streams to update FileSystem.Statistics counters on IO.

2020-04-28 Thread GitBox
mehakmeet commented on pull request #1918: URL: https://github.com/apache/hadoop/pull/1918#issuecomment-620434443 @amarnathkarthik it fails when you run it alone too ? ``` mvn -T 1C -Dparallel-tests=abfs clean verify -Dtest=none -Dit.test=ITestAbfsStreamStatistics ``` Never

[GitHub] [hadoop] hadoop-yetus commented on pull request #1950: HADOOP-16586. ITestS3GuardFsck, others fails when run using a local m…

2020-04-28 Thread GitBox
hadoop-yetus commented on pull request #1950: URL: https://github.com/apache/hadoop/pull/1950#issuecomment-620431057 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m

[GitHub] [hadoop] iwasakims commented on a change in pull request #1950: HADOOP-16586. ITestS3GuardFsck, others fails when run using a local m…

2020-04-28 Thread GitBox
iwasakims commented on a change in pull request #1950: URL: https://github.com/apache/hadoop/pull/1950#discussion_r416354605 ## File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardToolLocal.java ## @@ -165,18 +165,19 @@ public void