[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991160#comment-16991160 ] Da Zhou commented on HADOOP-16612: -- [~jeeteshm] committed to 3.2 > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > Attachments: HADOOP-16612.001.patch, HADOOP-16612.002.patch, > HADOOP-16612.003.patch, HADOOP-16612.004.patch > > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency > in the Hadoop ABFS driver. > The latency information is sent back to the ADLS Gen 2 REST API endpoints in > the subsequent requests. > Here's the PR: https://github.com/apache/hadoop/pull/1611 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16981892#comment-16981892 ] Jeetesh Mangwani commented on HADOOP-16612: --- [~DanielZhou] Yes, please include this change in branch 3.2 as well. > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > Attachments: HADOOP-16612.001.patch, HADOOP-16612.002.patch, > HADOOP-16612.003.patch, HADOOP-16612.004.patch > > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency > in the Hadoop ABFS driver. > The latency information is sent back to the ADLS Gen 2 REST API endpoints in > the subsequent requests. > Here's the PR: https://github.com/apache/hadoop/pull/1611 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977659#comment-16977659 ] Hudson commented on HADOOP-16612: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17667 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17667/]) HADOOP-16612. Track Azure Blob File System client-perceived latency (dazhou: rev b033c681e4fc3ee1a38caa807e130aee481d99d5) * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java * (edit) hadoop-tools/hadoop-azure/src/site/markdown/abfs.md * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsClient.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/HttpHeaderConfigurations.java * (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/AbfsPerfLoggable.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java * (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsPerfInfo.java * (edit) hadoop-tools/hadoop-azure/pom.xml * (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsPerfTracker.java * (edit) hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsPerfTracker.java > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > Attachments: HADOOP-16612.001.patch, HADOOP-16612.002.patch, > HADOOP-16612.003.patch, HADOOP-16612.004.patch > > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency > in the Hadoop ABFS driver. > The latency information is sent back to the ADLS Gen 2 REST API endpoints in > the subsequent requests. > Here's the PR: https://github.com/apache/hadoop/pull/1611 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977651#comment-16977651 ] Da Zhou commented on HADOOP-16612: -- [~jeeteshm] your change is committed into the trunk, let me know if you need it to be in branch-3.2 too. > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > Attachments: HADOOP-16612.001.patch, HADOOP-16612.002.patch, > HADOOP-16612.003.patch, HADOOP-16612.004.patch > > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency > in the Hadoop ABFS driver. > The latency information is sent back to the ADLS Gen 2 REST API endpoints in > the subsequent requests. > Here's the PR: https://github.com/apache/hadoop/pull/1611 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974609#comment-16974609 ] Jeetesh Mangwani commented on HADOOP-16612: --- Here are the test results for the patch https://issues.apache.org/jira/secure/attachment/12985323/HADOOP-16612.004.patch: -- non-xns: [INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 395, Failures: 0, Errors: 0, Skipped: 207 [WARNING] Tests run: 192, Failures: 0, Errors: 0, Skipped: 24 xns: [INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Failures: [ERROR] ITestGetNameSpaceEnabled.testXNSAccount:51->Assert.assertTrue:41->Assert.fail:88 Expecting getIsNamespaceEnabled() return true [INFO] [ERROR] Tests run: 395, Failures: 1, Errors: 0, Skipped: 207 [WARNING] Tests run: 192, Failures: 0, Errors: 0, Skipped: 24 - Comments: 1. ITestGetNameSpaceEnabled.testNonXNSAccount: fails because the HTTP response status is not 400, but is 404 > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > Attachments: HADOOP-16612.001.patch, HADOOP-16612.002.patch, > HADOOP-16612.003.patch, HADOOP-16612.004.patch > > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency > in the Hadoop ABFS driver. > The latency information is sent back to the ADLS Gen 2 REST API endpoints in > the subsequent requests. > Here's the PR: https://github.com/apache/hadoop/pull/1611 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[
https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974601#comment-16974601
]
Hadoop QA commented on HADOOP-16612:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HADOOP-16612 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985886/HADOOP-16612.004.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 2920f9448104 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d0302d3 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16670/testReport/ |
| Max. process+thread count | 309 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16670/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Track Azure
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972690#comment-16972690 ] Íñigo Goiri commented on HADOOP-16612: -- +1 (I also approved the PR). In my opinion, we should enable checkstyle for this build. > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > Attachments: HADOOP-16612.001.patch, HADOOP-16612.002.patch, > HADOOP-16612.003.patch > > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency > in the Hadoop ABFS driver. > The latency information is sent back to the ADLS Gen 2 REST API endpoints in > the subsequent requests. > Here's the PR: https://github.com/apache/hadoop/pull/1611 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970578#comment-16970578 ] Jeetesh Mangwani commented on HADOOP-16612: --- Test results: non-xns [INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 395, Failures: 0, Errors: 0, Skipped: 207 [ERROR] Tests run: 192, Failures: 0, Errors: 2, Skipped: 24 [ERROR] Errors: [ERROR] ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:60->testReadWriteAndSeek:75 » TestTimedOut [ERROR] ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:77 » TestTimedOut xns [INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Tests run: 395, Failures: 1, Errors: 0, Skipped: 207 [ERROR] Failures: [ERROR] ITestGetNameSpaceEnabled.testXNSAccount:51->Assert.assertTrue:41->Assert.fail:88 Expecting getIsNamespaceEnabled() return true [WARNING] Tests run: 192, Failures: 0, Errors: 0, Skipped: 24 --- Comments: 1. ITestGetNameSpaceEnabled.testNonXNSAccount: fails because the HTTP response status is not 400, but is 404 3. ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek: times out, probably because this is a scale test and my VM is slow 4. ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads: times out, probably because there are lot of heavy writes and my VM is slow > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > Attachments: HADOOP-16612.001.patch, HADOOP-16612.002.patch, > HADOOP-16612.003.patch > > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency > in the Hadoop ABFS driver. > The latency information is sent back to the ADLS Gen 2 REST API endpoints in > the subsequent requests. > Here's the PR: https://github.com/apache/hadoop/pull/1611 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[
https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969918#comment-16969918
]
Hadoop QA commented on HADOOP-16612:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HADOOP-16612 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985323/HADOOP-16612.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 1dd94c5f94c8 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 42fc888 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16650/testReport/ |
| Max. process+thread count | 312 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16650/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Track Azure
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[
https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969754#comment-16969754
]
Hadoop QA commented on HADOOP-16612:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 16s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 2
new + 5 unchanged - 0 fixed = 7 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
16s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HADOOP-16612 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985293/HADOOP-16612.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 48056d99ed4e 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 247584e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16646/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16646/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[
https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969685#comment-16969685
]
Hadoop QA commented on HADOOP-16612:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red} HADOOP-16612 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-16612 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985291/HADOOP-16612.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16645/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Track Azure Blob File System client-perceived latency
> -
>
> Key: HADOOP-16612
> URL: https://issues.apache.org/jira/browse/HADOOP-16612
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, hdfs-client
>Reporter: Jeetesh Mangwani
>Assignee: Jeetesh Mangwani
>Priority: Major
> Attachments: HADOOP-16612-001.patch, HADOOP-16612.001.patch
>
>
> Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency
> in the Hadoop ABFS driver.
> The latency information is sent back to the ADLS Gen 2 REST API endpoints in
> the subsequent requests.
> Here's the PR: https://github.com/apache/hadoop/pull/1611
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[
https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969679#comment-16969679
]
Hadoop QA commented on HADOOP-16612:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red} HADOOP-16612 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-16612 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985290/HADOOP-16612-001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16644/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Track Azure Blob File System client-perceived latency
> -
>
> Key: HADOOP-16612
> URL: https://issues.apache.org/jira/browse/HADOOP-16612
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, hdfs-client
>Reporter: Jeetesh Mangwani
>Assignee: Jeetesh Mangwani
>Priority: Major
> Attachments: HADOOP-16612-001.patch
>
>
> Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency
> in the Hadoop ABFS driver.
> The latency information is sent back to the ADLS Gen 2 REST API endpoints in
> the subsequent requests.
> Here's the PR: https://github.com/apache/hadoop/pull/1611
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[
https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969670#comment-16969670
]
Hadoop QA commented on HADOOP-16612:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red} HADOOP-16612 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-16612 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985289/1611.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16643/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Track Azure Blob File System client-perceived latency
> -
>
> Key: HADOOP-16612
> URL: https://issues.apache.org/jira/browse/HADOOP-16612
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, hdfs-client
>Reporter: Jeetesh Mangwani
>Assignee: Jeetesh Mangwani
>Priority: Major
> Attachments: 1611.patch
>
>
> Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency
> in the Hadoop ABFS driver.
> The latency information is sent back to the ADLS Gen 2 REST API endpoints in
> the subsequent requests.
> Here's the PR: https://github.com/apache/hadoop/pull/1611
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[
https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969649#comment-16969649
]
Hadoop QA commented on HADOOP-16612:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 13s{color}
| {color:red} HADOOP-16612 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-16612 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985285/1611.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16642/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Track Azure Blob File System client-perceived latency
> -
>
> Key: HADOOP-16612
> URL: https://issues.apache.org/jira/browse/HADOOP-16612
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, hdfs-client
>Reporter: Jeetesh Mangwani
>Assignee: Jeetesh Mangwani
>Priority: Major
> Attachments: 1611.patch
>
>
> Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency
> in the Hadoop ABFS driver.
> The latency information is sent back to the ADLS Gen 2 REST API endpoints in
> the subsequent requests.
> Here's the PR: https://github.com/apache/hadoop/pull/1611
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969591#comment-16969591 ] Da Zhou commented on HADOOP-16612: -- [~jeeteshm]I don't know how to force the Yetus to generate report for the PR, normally it is generated automatically after your update. Another alternative I can think of is to submit the patch under this JIRA and get the yetus support. > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency > in the Hadoop ABFS driver. > The latency information is sent back to the ADLS Gen 2 REST API endpoints in > the subsequent requests. > Here's the PR: https://github.com/apache/hadoop/pull/1611 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969544#comment-16969544 ] Jeetesh Mangwani commented on HADOOP-16612: --- [~ste...@apache.org], [~DanielZhou], [~billie] & [~inigoiri]: The Hadoop Yetus seems to have lost track of my pull request. I don't see checkstyle scan for the changes I pushed last. Does anyone know how I can force Yetus to run a validation build for my PR? > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency > in the Hadoop ABFS driver. > The latency information is sent back to the ADLS Gen 2 REST API endpoints in > the subsequent requests. > Here's the PR: https://github.com/apache/hadoop/pull/1611 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957192#comment-16957192 ] Billie Rinaldi commented on HADOOP-16612: - bq. Does the Hadoop's tracing system provide for sending latency info back to the ADLS server? Sure, the tracing information can be sent anywhere with a custom SpanReceiver implementation, though it might be getting all the traces and not just the ABFS ones. bq. HTrace is EOL/unsupported. We're still working out what to do next there Yes, I see there is a [ticket|https://jira.apache.org/jira/browse/HADOOP-15566] open to switch to opentracing. All the tracing instrumentation libraries are pretty similar to each other, so it won't make much of a difference. I am okay with proceeding with this x-ms-abfs-client-latency header approach. I wanted to make sure people are aware of the tracing possibility since it is a good way to collect timing information, and we may want to consider instrumenting the ABFS driver for it in the future. > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency > in the Hadoop ABFS driver. > The latency information is sent back to the ADLS Gen 2 REST API endpoints in > the subsequent requests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957144#comment-16957144 ] Steve Loughran commented on HADOOP-16612: - Billie, HTrace is EOL/unsupported. We're still working out what to do next there > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency > in the Hadoop ABFS driver. > The latency information is sent back to the ADLS Gen 2 REST API endpoints in > the subsequent requests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956571#comment-16956571 ] Jeetesh Mangwani commented on HADOOP-16612: --- Hi [~DanielZhou], After syncing with the latest trunk and correcting some of my storage accounts, most of the errors are gone. However, I still see failures on trunk and my feature branch as well. I think the failing test will require a separate fix. Can we address this in a separate PR? trunk, non-xns [INFO] Tests run: 42, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 395, Failures: 0, Errors: 0, Skipped: 207 [WARNING] Tests run: 192, Failures: 0, Errors: 0, Skipped: 24 = feature branch, non-xns [INFO] Tests run: 53, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 395, Failures: 0, Errors: 0, Skipped: 207 [WARNING] Tests run: 192, Failures: 0, Errors: 0, Skipped: 24 = trunk, xns [INFO] Tests run: 42, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemCLI>AbstractAbfsIntegrationTest.setup:137 » AbfsRestOperation [INFO] [ERROR] Tests run: 395, Failures: 0, Errors: 1, Skipped: 21 [WARNING] Tests run: 192, Failures: 0, Errors: 0, Skipped: 24 feature branch, xns [INFO] Tests run: 53, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemCLI>AbstractAbfsIntegrationTest.setup:137 » AbfsRestOperation [INFO] [ERROR] Tests run: 395, Failures: 0, Errors: 1, Skipped: 21 [WARNING] Tests run: 192, Failures: 0, Errors: 0, Skipped: 24 Stack trace of the failing test: [ERROR] testMkdirRootNonExistentContainer(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCLI) Time elapsed: 12.776 s <<< ERROR! Operation failed: "The specified filesystem does not exist.", 404, HEAD, https://abfstest01.dfs.core.windows.net/abfs-testcontainer-4377d2f0-8f6e-4e0d-8a9a-f35c2f48bac7//?upn=false&action=getAccessControl&timeout=90 at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:135) at org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:516) at org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:499) at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getIsNamespaceEnabled(AzureBlobFileSystemStore.java:216) at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getIsNamespaceEnabled(AzureBlobFileSystem.java:1108) at org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.setup(AbstractAbfsIntegrationTest.java:137) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency > in the Hadoop ABFS driver. > The latency information is sent back to the ADLS Gen 2 REST API endpoints in > the subsequent requests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956565#comment-16956565 ] Jeetesh Mangwani commented on HADOOP-16612: --- Thank you [~billie] for your attention. After reading through the doc referred by you, I inferred that Hadoop's tracing system doesn't serve the requirements set down for this JIRA. We wish to send the client-side latency information back to the ADLS Gen 2 endpoint. This helps us track the end-to-end latency together with the latency metrics of ADLS' internal sub-systems. Does the Hadoop's tracing system provide for sending latency info back to the ADLS server? > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency > in the Hadoop ABFS driver. > The latency information is sent back to the ADLS Gen 2 REST API endpoints in > the subsequent requests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954848#comment-16954848 ] Da Zhou commented on HADOOP-16612: -- Hi [~jeeteshm] could you sync with the latest trunk? the first two test failures with status code should be fixed with this commit [https://github.com/apache/hadoop/pull/1498] : You can ignore the timeout for your scale tests for now. > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring > latencies in the Hadoop ABFS driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954832#comment-16954832 ] Billie Rinaldi commented on HADOOP-16612: - Thanks for working on this issue, [~jeeteshm]. Have you evaluated instrumenting the ABFS driver with Hadoop's [existing tracing system|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Tracing.html] instead of (or in addition to) creating a new instrumentation system? > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring > latencies in the Hadoop ABFS driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949782#comment-16949782 ] Jeetesh Mangwani commented on HADOOP-16612: --- I have fixed the checkstyle issues. Please take a look. When I run tests on my machine, I see 4 failing tests on my 'trunk' branch. These tests fail on my feature branch too. 1. ITestGetNameSpaceEnabled.testNonXNSAccount: fails because the HTTP response status is not 400, but is 404 2. ITestAzureBlobFileSystemCLI.testMkdirRootNonExistentContainer: fails in the setup phase because the HTTP response status is not 400, but is 404 3. ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek: times out, probably because this is a scale test and my machine is slow 4. ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads: times out, probably because there are lot of heavy writes and my machine is slow Here are the details: trunk, non-xns [INFO] Tests run: 42, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Failures: [ERROR] ITestGetNameSpaceEnabled.testNonXNSAccount:59->Assert.assertFalse:64->Assert.assertTrue:41->Assert.fail:88 Expecting getIsNamespaceEnabled() return false [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemCLI>AbstractAbfsIntegrationTest.setup:137 » AbfsRestOperation [INFO] [ERROR] Tests run: 395, Failures: 1, Errors: 1, Skipped: 21 [ERROR] Errors: [ERROR] ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:60->testReadWriteAndSeek:75 » TestTimedOut [ERROR] ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:77 » TestTimedOut [INFO] [ERROR] Tests run: 192, Failures: 0, Errors: 2, Skipped: 24 === feature branch, non-xns [INFO] Tests run: 53, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Failures: [ERROR] ITestGetNameSpaceEnabled.testNonXNSAccount:59->Assert.assertFalse:64->Assert.assertTrue:41->Assert.fail:88 Expecting getIsNamespaceEnabled() return false [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemCLI>AbstractAbfsIntegrationTest.setup:137 » AbfsRestOperation [INFO] [ERROR] Tests run: 395, Failures: 1, Errors: 1, Skipped: 21 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:60->testReadWriteAndSeek:75 » TestTimedOut [ERROR] ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:77 » TestTimedOut [INFO] [ERROR] Tests run: 192, Failures: 0, Errors: 2, Skipped: 24 trunk, xns [INFO] Tests run: 42, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemCLI>AbstractAbfsIntegrationTest.setup:137 » AbfsRestOperation [INFO] [ERROR] Tests run: 395, Failures: 0, Errors: 1, Skipped: 21 [ERROR] Errors: [ERROR] ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:60->testReadWriteAndSeek:75 » TestTimedOut [ERROR] ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:77 » TestTimedOut [INFO] [ERROR] Tests run: 192, Failures: 0, Errors: 2, Skipped: 24 = feature branch, xns [INFO] Tests run: 53, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemCLI>AbstractAbfsIntegrationTest.setup:137 » AbfsRestOperation [INFO] [ERROR] Tests run: 395, Failures: 0, Errors: 1, Skipped: 21 [ERROR] Errors: [ERROR] ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:60->testReadWriteAndSeek:75 » TestTimedOut [ERROR] ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:77 » TestTimedOut [INFO] [ERROR] Tests run: 192, Failures: 0, Errors: 2, Skipped: 24 Error stack traces: [ERROR] testMkdirRootNonExistentContainer(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCLI) Time elapsed: 18.27 s <<< ERROR! Operation failed: "The specified filesystem does not exist.", 404, HEAD, https://abfstest02.dfs.core.windows.net/abfs-testcontainer-aa478873-647e-455a-9a71-4cb6da30a088//?upn=false&action=getAccessControl&timeout=90 at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:143) at org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:523) at org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:506) at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getIsNamespaceEnabled(AzureBlobFileSystemStore.java:224) at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getIsNamespaceEnabled(AzureBlobFileSystem.java:1108) at org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.setup(AbstractAbfsIntegrationTest.java:137) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMet
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16943739#comment-16943739 ] Da Zhou commented on HADOOP-16612: -- please fix the checkstyle issues and share the hadoop azure tests results here, [check |https://issues.apache.org/jira/browse/HADOOP-16578?focusedCommentId=16939244&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16939244]this for example > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring > latencies in the Hadoop ABFS driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942356#comment-16942356 ] Jeetesh Mangwani commented on HADOOP-16612: --- Here's the PR: https://github.com/apache/hadoop/pull/1569 > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring > latencies in the Hadoop ABFS driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941104#comment-16941104 ] Da Zhou commented on HADOOP-16612: -- Could you open a GitHub PR for this JIRA? that will make it easy for others to review too. > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring > latencies in the Hadoop ABFS driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[
https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939884#comment-16939884
]
Hadoop QA commented on HADOOP-16612:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 16s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated
71 new + 5 unchanged - 0 fixed = 76 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:efed4450bf1 |
| JIRA Issue | HADOOP-16612 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981625/HADOOP-16612.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 68f918fef6f9 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 14b4fbc |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16554/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16554/testReport/ |
| Max. process+thread count | 308 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-a
