[jira] [Assigned] (HADOOP-14269) Create module-info.java for each module
[ https://issues.apache.org/jira/browse/HADOOP-14269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal reassigned HADOOP-14269: --- Assignee: (was: Adam Antal) > Create module-info.java for each module > --- > > Key: HADOOP-14269 > URL: https://issues.apache.org/jira/browse/HADOOP-14269 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Priority: Major > > module-info.java is required for JDK9 Jigsaw feature. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14269) Create module-info.java for each module
[ https://issues.apache.org/jira/browse/HADOOP-14269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal reassigned HADOOP-14269: --- Assignee: Adam Antal > Create module-info.java for each module > --- > > Key: HADOOP-14269 > URL: https://issues.apache.org/jira/browse/HADOOP-14269 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Adam Antal >Priority: Major > > module-info.java is required for JDK9 Jigsaw feature. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988721#comment-16988721 ] Adam Antal commented on HADOOP-16683: - Hi [~snemeth], The branch-3.2 and branch-3.1 patches have green jenkins build, could you please backport so we can close this jira? > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16683.001.patch, HADOOP-16683.002.patch, > HADOOP-16683.003.patch, HADOOP-16683.branch-3.1.001.patch, > HADOOP-16683.branch-3.2.001.patch, HADOOP-16683.branch-3.2.001.patch > > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16729) Extract version numbers to head of pom.xml
[ https://issues.apache.org/jira/browse/HADOOP-16729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983473#comment-16983473 ] Adam Antal commented on HADOOP-16729: - Maybe this could be a subtask of HADOOP-9991. > Extract version numbers to head of pom.xml > -- > > Key: HADOOP-16729 > URL: https://issues.apache.org/jira/browse/HADOOP-16729 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Tamas Penzes >Assignee: Tamas Penzes >Priority: Minor > > To be able to easily replace third-party dependency version numbers it would > be useful to collect their version numbers at the top of the pom.xml. > For many third-parties (e.g. slf4j, jetty, etc.) this is already done, but I > would need the same for others. The change doesn't have any effect on the > code or the build, no version numbers would be changed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16539) ABFS: Add missing query parameter for getPathStatus
[ https://issues.apache.org/jira/browse/HADOOP-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated HADOOP-16539: Status: Patch Available (was: Open) > ABFS: Add missing query parameter for getPathStatus > --- > > Key: HADOOP-16539 > URL: https://issues.apache.org/jira/browse/HADOOP-16539 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Adam Antal >Priority: Major > Attachments: HADOOP-16539.001.patch > > > When calling > [getPathStatus|https://github.com/apache/hadoop/blob/e220dac15cc9972ebdd54ea9c82f288f234fca51/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java#L356], > query parameter "action=getStatus" is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16539) ABFS: Add missing query parameter for getPathStatus
[ https://issues.apache.org/jira/browse/HADOOP-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated HADOOP-16539: Attachment: HADOOP-16539.001.patch > ABFS: Add missing query parameter for getPathStatus > --- > > Key: HADOOP-16539 > URL: https://issues.apache.org/jira/browse/HADOOP-16539 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Adam Antal >Priority: Major > Attachments: HADOOP-16539.001.patch > > > When calling > [getPathStatus|https://github.com/apache/hadoop/blob/e220dac15cc9972ebdd54ea9c82f288f234fca51/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java#L356], > query parameter "action=getStatus" is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16539) ABFS: Add missing query parameter for getPathStatus
[ https://issues.apache.org/jira/browse/HADOOP-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal reassigned HADOOP-16539: --- Assignee: Adam Antal > ABFS: Add missing query parameter for getPathStatus > --- > > Key: HADOOP-16539 > URL: https://issues.apache.org/jira/browse/HADOOP-16539 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Adam Antal >Priority: Major > > When calling > [getPathStatus|https://github.com/apache/hadoop/blob/e220dac15cc9972ebdd54ea9c82f288f234fca51/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java#L356], > query parameter "action=getStatus" is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975170#comment-16975170 ] Adam Antal commented on HADOOP-16683: - Thanks [~pbacsko]! Reuploaded the patch for branch-3.2. Could you please backport these when jenkins finished [~snemeth]? Thanks! > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16683.001.patch, HADOOP-16683.002.patch, > HADOOP-16683.003.patch, HADOOP-16683.branch-3.1.001.patch, > HADOOP-16683.branch-3.2.001.patch, HADOOP-16683.branch-3.2.001.patch > > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated HADOOP-16683: Attachment: HADOOP-16683.branch-3.2.001.patch > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16683.001.patch, HADOOP-16683.002.patch, > HADOOP-16683.003.patch, HADOOP-16683.branch-3.1.001.patch, > HADOOP-16683.branch-3.2.001.patch, HADOOP-16683.branch-3.2.001.patch > > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16713) Use PathCapabilities for default configuring append mode for RollingFileSystemSink
Adam Antal created HADOOP-16713:
---
Summary: Use PathCapabilities for default configuring append mode
for RollingFileSystemSink
Key: HADOOP-16713
URL: https://issues.apache.org/jira/browse/HADOOP-16713
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Affects Versions: 3.3.0
Reporter: Adam Antal
{{RollingFileSystemSink}} uses a filesystem to store metrics. The key
{{allow-append}} is disabled by default, but if enabled, new metrics can be
appended to an existing file.
Given that we can have the {{PathCapabilities}} interface, we can change the
default of {{allow-append}} mode depending on the support of the append
operation decided by the {{FileSystem.hasPathCapability()}} call
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated HADOOP-16683: Attachment: HADOOP-16683.branch-3.1.001.patch > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16683.001.patch, HADOOP-16683.002.patch, > HADOOP-16683.003.patch, HADOOP-16683.branch-3.1.001.patch, > HADOOP-16683.branch-3.2.001.patch > > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated HADOOP-16683: Attachment: HADOOP-16683.branch-3.2.001.patch > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16683.001.patch, HADOOP-16683.002.patch, > HADOOP-16683.003.patch, HADOOP-16683.branch-3.2.001.patch > > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal reopened HADOOP-16683: - Let's backport this issue to lower branches. > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16683.001.patch, HADOOP-16683.002.patch, > HADOOP-16683.003.patch > > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974304#comment-16974304 ] Adam Antal commented on HADOOP-16683: - Hi [~xkrogen]! Yes, it would make sense. Let me reopen this jira and upload two patch for branch-3.2 and branch-3.1. > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16683.001.patch, HADOOP-16683.002.patch, > HADOOP-16683.003.patch > > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated HADOOP-16683: Target Version/s: 3.3.0, 3.1.4, 3.2.2 (was: 3.3.0) Status: Patch Available (was: Reopened) > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16683.001.patch, HADOOP-16683.002.patch, > HADOOP-16683.003.patch > > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971486#comment-16971486 ] Adam Antal commented on HADOOP-16683: - Thanks for the commit [~snemeth]. > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16683.001.patch, HADOOP-16683.002.patch, > HADOOP-16683.003.patch > > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated HADOOP-16683: Attachment: HADOOP-16683.003.patch > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: HADOOP-16683.001.patch, HADOOP-16683.002.patch, > HADOOP-16683.003.patch > > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968323#comment-16968323 ] Adam Antal commented on HADOOP-16683: - The RM and NM clients usually wrap it around an IOException I think, or at least this is what I encountered. I can see rationale in more aggressive guarding, so I'll search for an AccessControlException along the chain. > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: HADOOP-16683.001.patch, HADOOP-16683.002.patch > > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968196#comment-16968196 ] Adam Antal commented on HADOOP-16683: - Nice suggestion [~pbacsko]. Fixed in patch v2. > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: HADOOP-16683.001.patch, HADOOP-16683.002.patch > > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated HADOOP-16683: Attachment: HADOOP-16683.002.patch > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: HADOOP-16683.001.patch, HADOOP-16683.002.patch > > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated HADOOP-16683: Status: Patch Available (was: In Progress) > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: HADOOP-16683.001.patch > > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated HADOOP-16683: Attachment: HADOOP-16683.001.patch > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: HADOOP-16683.001.patch > > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16683 started by Adam Antal. --- > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16580) Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException
[
https://issues.apache.org/jira/browse/HADOOP-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967616#comment-16967616
]
Adam Antal commented on HADOOP-16580:
-
Seems like we didn't cover all the cases. Filed HADOOP-16683 for the wrapped
case.
> Disable retry of FailoverOnNetworkExceptionRetry in case of
> AccessControlException
> --
>
> Key: HADOOP-16580
> URL: https://issues.apache.org/jira/browse/HADOOP-16580
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HADOOP-16580.001.patch, HADOOP-16580.002.patch,
> HADOOP-16580.003.patch, HADOOP-16580.branch-3.2.001.patch
>
>
> HADOOP-14982 handled the case where a SaslException is thrown. The issue
> still persists, since the exception that is thrown is an
> *AccessControlException* because user has no kerberos credentials.
> My suggestion is that we should add this case as well to
> {{FailoverOnNetworkExceptionRetry}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
Adam Antal created HADOOP-16683: --- Summary: Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException Key: HADOOP-16683 URL: https://issues.apache.org/jira/browse/HADOOP-16683 Project: Hadoop Common Issue Type: Bug Components: common Affects Versions: 3.3.0 Reporter: Adam Antal Assignee: Adam Antal Follow up patch on HADOOP-16580. We successfully disabled the retry in case of an AccessControlException which has resolved some of the cases, but in other cases AccessControlException is wrapped inside another IOException and you can only get the original exception by calling getCause(). Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12693) Many misusages of assertEquals(expected, actual)
[
https://issues.apache.org/jira/browse/HADOOP-12693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966854#comment-16966854
]
Adam Antal commented on HADOOP-12693:
-
Common, Yarn, MR and Tools have been cleaned up, waiting for the HDFS part of
the job to finish.
> Many misusages of assertEquals(expected, actual)
>
>
> Key: HADOOP-12693
> URL: https://issues.apache.org/jira/browse/HADOOP-12693
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Akihiro Suda
>Priority: Trivial
> Attachments: just-rough-approx.txt
>
>
> The first arg of {{org.JUnit.Assert.assertEquals()}} should be an
> {{expected}} value, and the second one should be an {{actual}} value.
> {code}
> void assertEquals(T expected, T actual);
> {code}
> http://junit.org/apidocs/org/junit/Assert.html#assertEquals(java.lang.Object,
> java.lang.Object)
> However, there are so many violations in Hadoop, which can make a misleading
> message like this:
> {code}
> AssertionError: expected: but was:
> {code}.
> Please refer to {{just-rough-approx.txt}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16510) [hadoop-common] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966852#comment-16966852
]
Adam Antal commented on HADOOP-16510:
-
The other issues are not backported to other branches than trunk, so I think
it's safe to resolve this issue.
> [hadoop-common] Fix order of actual and expected expression in assert
> statements
>
>
> Key: HADOOP-16510
> URL: https://issues.apache.org/jira/browse/HADOOP-16510
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16510.001.patch, HADOOP-16510.002.patch,
> HADOOP-16510.003.patch
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up jira for the hadoop-common project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16510) [hadoop-common] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16510:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> [hadoop-common] Fix order of actual and expected expression in assert
> statements
>
>
> Key: HADOOP-16510
> URL: https://issues.apache.org/jira/browse/HADOOP-16510
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16510.001.patch, HADOOP-16510.002.patch,
> HADOOP-16510.003.patch
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up jira for the hadoop-common project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16510) [hadoop-common] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966850#comment-16966850
]
Adam Antal commented on HADOOP-16510:
-
Thanks for committing this [~snemeth]!
> [hadoop-common] Fix order of actual and expected expression in assert
> statements
>
>
> Key: HADOOP-16510
> URL: https://issues.apache.org/jira/browse/HADOOP-16510
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16510.001.patch, HADOOP-16510.002.patch,
> HADOOP-16510.003.patch
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up jira for the hadoop-common project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16580) Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException
[
https://issues.apache.org/jira/browse/HADOOP-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952905#comment-16952905
]
Adam Antal commented on HADOOP-16580:
-
Thanks everyone for the reviews and [~snemeth] for the commit!
> Disable retry of FailoverOnNetworkExceptionRetry in case of
> AccessControlException
> --
>
> Key: HADOOP-16580
> URL: https://issues.apache.org/jira/browse/HADOOP-16580
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HADOOP-16580.001.patch, HADOOP-16580.002.patch,
> HADOOP-16580.003.patch, HADOOP-16580.branch-3.2.001.patch
>
>
> HADOOP-14982 handled the case where a SaslException is thrown. The issue
> still persists, since the exception that is thrown is an
> *AccessControlException* because user has no kerberos credentials.
> My suggestion is that we should add this case as well to
> {{FailoverOnNetworkExceptionRetry}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16510) [hadoop-common] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951864#comment-16951864
]
Adam Antal commented on HADOOP-16510:
-
Cannot reproduce the TestFixKerberosTicketOrder.test failure locally - I
suppose this is some intermittent issue.
The javac error is still not relevant since the mentioned deprecated method was
present before the patch and I simply didn't remove it.
> [hadoop-common] Fix order of actual and expected expression in assert
> statements
>
>
> Key: HADOOP-16510
> URL: https://issues.apache.org/jira/browse/HADOOP-16510
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16510.001.patch, HADOOP-16510.002.patch,
> HADOOP-16510.003.patch
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up jira for the hadoop-common project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16580) Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException
[
https://issues.apache.org/jira/browse/HADOOP-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951859#comment-16951859
]
Adam Antal commented on HADOOP-16580:
-
One last checkstyle should be ignored because the surrounding code is also
misindented.
If you agree with the javadocs could you please commit this [~snemeth]?
> Disable retry of FailoverOnNetworkExceptionRetry in case of
> AccessControlException
> --
>
> Key: HADOOP-16580
> URL: https://issues.apache.org/jira/browse/HADOOP-16580
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16580.001.patch, HADOOP-16580.002.patch,
> HADOOP-16580.003.patch
>
>
> HADOOP-14982 handled the case where a SaslException is thrown. The issue
> still persists, since the exception that is thrown is an
> *AccessControlException* because user has no kerberos credentials.
> My suggestion is that we should add this case as well to
> {{FailoverOnNetworkExceptionRetry}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16580) Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException
[
https://issues.apache.org/jira/browse/HADOOP-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951079#comment-16951079
]
Adam Antal commented on HADOOP-16580:
-
Thanks for the review [~snemeth]. I have uploaded patchset v3 which added
javadocs to the classes that are affected by this patch.
> Disable retry of FailoverOnNetworkExceptionRetry in case of
> AccessControlException
> --
>
> Key: HADOOP-16580
> URL: https://issues.apache.org/jira/browse/HADOOP-16580
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16580.001.patch, HADOOP-16580.002.patch,
> HADOOP-16580.003.patch
>
>
> HADOOP-14982 handled the case where a SaslException is thrown. The issue
> still persists, since the exception that is thrown is an
> *AccessControlException* because user has no kerberos credentials.
> My suggestion is that we should add this case as well to
> {{FailoverOnNetworkExceptionRetry}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16580) Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException
[
https://issues.apache.org/jira/browse/HADOOP-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16580:
Attachment: HADOOP-16580.003.patch
> Disable retry of FailoverOnNetworkExceptionRetry in case of
> AccessControlException
> --
>
> Key: HADOOP-16580
> URL: https://issues.apache.org/jira/browse/HADOOP-16580
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16580.001.patch, HADOOP-16580.002.patch,
> HADOOP-16580.003.patch
>
>
> HADOOP-14982 handled the case where a SaslException is thrown. The issue
> still persists, since the exception that is thrown is an
> *AccessControlException* because user has no kerberos credentials.
> My suggestion is that we should add this case as well to
> {{FailoverOnNetworkExceptionRetry}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16510) [hadoop-common] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16510:
Attachment: HADOOP-16510.003.patch
> [hadoop-common] Fix order of actual and expected expression in assert
> statements
>
>
> Key: HADOOP-16510
> URL: https://issues.apache.org/jira/browse/HADOOP-16510
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16510.001.patch, HADOOP-16510.002.patch,
> HADOOP-16510.003.patch
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up jira for the hadoop-common project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16510) [hadoop-common] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951044#comment-16951044
]
Adam Antal commented on HADOOP-16510:
-
Javac error is irrelevant as I didn't add that call, just rewrite the assertion
surrounding it.
Fixed last checkstyle in [^HADOOP-16510.003.patch].
> [hadoop-common] Fix order of actual and expected expression in assert
> statements
>
>
> Key: HADOOP-16510
> URL: https://issues.apache.org/jira/browse/HADOOP-16510
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16510.001.patch, HADOOP-16510.002.patch,
> HADOOP-16510.003.patch
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up jira for the hadoop-common project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16510) [hadoop-common] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950908#comment-16950908
]
Adam Antal commented on HADOOP-16510:
-
Uploaded patchset v2 - fixing checkstyle and failed tests.
Also, a short notice to everyone: patch v1 introduced a timeout in
{{TestLightWeightResizableGSet}} because {{assertThat$contains}} does not
default to {{Iterable$contains}}, it uses a stream-filter on
object-collect-assertNotEmpty solution which is *very* inefficient (thus the
timeout) - also that would not test the behaviour of the
{{LightWeightResizableGSet}} anyways. Fixed all of this in patchset v2.
> [hadoop-common] Fix order of actual and expected expression in assert
> statements
>
>
> Key: HADOOP-16510
> URL: https://issues.apache.org/jira/browse/HADOOP-16510
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16510.001.patch, HADOOP-16510.002.patch
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up jira for the hadoop-common project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16510) [hadoop-common] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16510:
Attachment: HADOOP-16510.002.patch
> [hadoop-common] Fix order of actual and expected expression in assert
> statements
>
>
> Key: HADOOP-16510
> URL: https://issues.apache.org/jira/browse/HADOOP-16510
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16510.001.patch, HADOOP-16510.002.patch
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up jira for the hadoop-common project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16510) [hadoop-common] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16510:
Status: Patch Available (was: In Progress)
> [hadoop-common] Fix order of actual and expected expression in assert
> statements
>
>
> Key: HADOOP-16510
> URL: https://issues.apache.org/jira/browse/HADOOP-16510
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16510.001.patch
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up jira for the hadoop-common project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16510) [hadoop-common] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16510:
Attachment: HADOOP-16510.001.patch
> [hadoop-common] Fix order of actual and expected expression in assert
> statements
>
>
> Key: HADOOP-16510
> URL: https://issues.apache.org/jira/browse/HADOOP-16510
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16510.001.patch
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up jira for the hadoop-common project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16511) [hadoop-hdfs] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945682#comment-16945682
]
Adam Antal commented on HADOOP-16511:
-
As discussed in the PR moving this to HDFS project.
[~vinayakumarb]: Could you please assign the jira user [~Kevin_Zheng] to the
HDFS project and assign this jira to him?
> [hadoop-hdfs] Fix order of actual and expected expression in assert statements
> --
>
> Key: HADOOP-16511
> URL: https://issues.apache.org/jira/browse/HADOOP-16511
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Priority: Major
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up jira for the hadoop-hdfs project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16511) [hadoop-hdfs] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16511:
Parent: (was: HADOOP-12693)
Issue Type: Bug (was: Sub-task)
> [hadoop-hdfs] Fix order of actual and expected expression in assert statements
> --
>
> Key: HADOOP-16511
> URL: https://issues.apache.org/jira/browse/HADOOP-16511
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Priority: Major
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up jira for the hadoop-hdfs project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942086#comment-16942086
]
Adam Antal commented on HADOOP-16503:
-
Thanks for sorting this out [~kmarton]!
> [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
> -
>
> Key: HADOOP-16503
> URL: https://issues.apache.org/jira/browse/HADOOP-16503
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Kinga Marton
>Priority: Major
> Attachments: HADOOP-16503.001.patch, HADOOP-16503.002.patch,
> HADOOP-16503.003.patch
>
>
> Many of the tests in
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
> fails with the following error message running on JDK11:
> {noformat}
> [ERROR]
> testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
> Time elapsed: 0.204 s <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> YarnConfiguration cannot be returned by getRMNodes()
> getRMNodes() should return ConcurrentMap
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
> spies -
>- with doReturn|Throw() family of methods. More in javadocs for
> Mockito.spy() method.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
>...
> {noformat}
> This is due to the actual execution of the call, while we need to record only
> the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941048#comment-16941048
]
Adam Antal commented on HADOOP-16503:
-
Thanks to the patch [~kmarton]. Could you please explain what does this patch
fix? You mentioned that YARN-9784 fixed the issue seen above, so I am unsure
what do we aim with the patch.
> [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
> -
>
> Key: HADOOP-16503
> URL: https://issues.apache.org/jira/browse/HADOOP-16503
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Kinga Marton
>Priority: Major
> Attachments: HADOOP-16503.001.patch, HADOOP-16503.002.patch,
> HADOOP-16503.003.patch
>
>
> Many of the tests in
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
> fails with the following error message running on JDK11:
> {noformat}
> [ERROR]
> testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
> Time elapsed: 0.204 s <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> YarnConfiguration cannot be returned by getRMNodes()
> getRMNodes() should return ConcurrentMap
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
> spies -
>- with doReturn|Throw() family of methods. More in javadocs for
> Mockito.spy() method.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
>...
> {noformat}
> This is due to the actual execution of the call, while we need to record only
> the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937719#comment-16937719
]
Adam Antal commented on HADOOP-15691:
-
Thanks for working on this [~ste...@apache.org]!
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch, HADOOP-15691-004.patch
>
>
> This complements the {{StreamCapabilities}} interface by allowing
> applications
> to probe for a specific path on a specific instance of a {{FileSystem}}
> or {{FileContext}} offering a specific feature.
> This is intended to allow applications to determine
> * Whether a method is implemented before calling it and dealing with
> any subsequent UnsupportedOperationException.
> * Whether a specific feature is believed to be available in the remote store.
> As well as a common set of capabilities defined in CommonPathCapabilities,
> file systems are free to add their own capabilities, prefixed with
> fs. + schema + .
>
> The plan is to identify and document more capabilities -and for file systems
> which add new features, for a declaration of the availability of the feature
> to
> always be available.
> The interface may be offered by other classes too; there is no restriction
> here.
> Note
> * The remote store is not expected to be checked for the feature;
> It is more a check of client API and the client's configuration/knowledge
> of the state of the remote system.
> * Permissions are not checked.
> This is needed for
> * HADOOP-14707: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16580) Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException
[
https://issues.apache.org/jira/browse/HADOOP-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936481#comment-16936481
]
Adam Antal commented on HADOOP-16580:
-
Can you take a look at this issue [~ste...@apache.org], [~sunilg]?
> Disable retry of FailoverOnNetworkExceptionRetry in case of
> AccessControlException
> --
>
> Key: HADOOP-16580
> URL: https://issues.apache.org/jira/browse/HADOOP-16580
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16580.001.patch, HADOOP-16580.002.patch
>
>
> HADOOP-14982 handled the case where a SaslException is thrown. The issue
> still persists, since the exception that is thrown is an
> *AccessControlException* because user has no kerberos credentials.
> My suggestion is that we should add this case as well to
> {{FailoverOnNetworkExceptionRetry}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16580) Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException
[
https://issues.apache.org/jira/browse/HADOOP-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932522#comment-16932522
]
Adam Antal commented on HADOOP-16580:
-
Thanks for the review [~shuzirra], let me give you a bit more clarification on
the patch.
UnreliableInterface is an interface for *purely testing purpose*. It is used
for testing RetryPolicy applied to the UnreliableInterface which mimics an
interface that is "unreliable" in a sense that it might throws exceptions. It
is used for checking that the RetryPolicy is triggered / works as expected
having an unreliable underlying resource (UnreliableImplementation) throwing
Exception in particular circumstances.
The annotation is something that is needed for the imitation - the API calls
(like {{mapred job -list}} - see HADOOP-14982) are usually annotated like that,
and in the code there's a part where:
{code:java}
...
} else if (e instanceof SocketException
|| (e instanceof IOException && !(e instanceof RemoteException))) {
if (isIdempotentOrAtMostOnce) {
return new RetryAction(RetryAction.RetryDecision.FAILOVER_AND_RETRY,
getFailoverOrRetrySleepTime(retries));
} else {
return new RetryAction(RetryAction.RetryDecision.FAIL, 0,
"the invoked method is not idempotent, and unable to determine "
+ "whether it was invoked");
}
} ...
{code}
where the {{isIdempotentOrAtMostOnce}} variable is determined by the
interface's annotation. Through reflection it is queried whether the method has
the {{@AtMostOnce}} or {{@Idempotent}} annotation and the
{{FailoverOnNetworkExceptionRetry}} RetryPolicy decides accordingly to retry or
fail.
Since I wanted to test the RetryPolicy on an Idempotent-annotated call (just
like the API call) which throws an AccessControlException I think the test is
logically correct, and the annotation is proper for this reason.
As for the name of the method:
The counter starts from 0, and each time the function is called it is
incremented by one and will throw exception each time until it reaches 8 (which
is the 9th call) where it won't emit any new exception. I think this is the
exact behaviour that the name suggests. If there's any other issue or adding
some javadoc to make this clearer I can work on that. It was hard to understand
me these concepts first since the lack of javadoc, but hope the above
description helps.
> Disable retry of FailoverOnNetworkExceptionRetry in case of
> AccessControlException
> --
>
> Key: HADOOP-16580
> URL: https://issues.apache.org/jira/browse/HADOOP-16580
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16580.001.patch, HADOOP-16580.002.patch
>
>
> HADOOP-14982 handled the case where a SaslException is thrown. The issue
> still persists, since the exception that is thrown is an
> *AccessControlException* because user has no kerberos credentials.
> My suggestion is that we should add this case as well to
> {{FailoverOnNetworkExceptionRetry}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16580) Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException
[
https://issues.apache.org/jira/browse/HADOOP-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932195#comment-16932195
]
Adam Antal commented on HADOOP-16580:
-
Patchset v2 takes care of one of the checkstyle issue - I'd rather disregard
the other one, as the surrounding block is also misindented.
Please review.
> Disable retry of FailoverOnNetworkExceptionRetry in case of
> AccessControlException
> --
>
> Key: HADOOP-16580
> URL: https://issues.apache.org/jira/browse/HADOOP-16580
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16580.001.patch, HADOOP-16580.002.patch
>
>
> HADOOP-14982 handled the case where a SaslException is thrown. The issue
> still persists, since the exception that is thrown is an
> *AccessControlException* because user has no kerberos credentials.
> My suggestion is that we should add this case as well to
> {{FailoverOnNetworkExceptionRetry}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16580) Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException
[
https://issues.apache.org/jira/browse/HADOOP-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16580:
Attachment: HADOOP-16580.002.patch
> Disable retry of FailoverOnNetworkExceptionRetry in case of
> AccessControlException
> --
>
> Key: HADOOP-16580
> URL: https://issues.apache.org/jira/browse/HADOOP-16580
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16580.001.patch, HADOOP-16580.002.patch
>
>
> HADOOP-14982 handled the case where a SaslException is thrown. The issue
> still persists, since the exception that is thrown is an
> *AccessControlException* because user has no kerberos credentials.
> My suggestion is that we should add this case as well to
> {{FailoverOnNetworkExceptionRetry}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16580) Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException
[
https://issues.apache.org/jira/browse/HADOOP-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931506#comment-16931506
]
Adam Antal commented on HADOOP-16580:
-
The solution is similar to HADOOP-14982. I suspect the call of {{mapred -list}}
and {{yarn logs}} both have {{Idempotent}} annotation that is responsible for
retry (otherwise in case of IOException like AccessControlException retry will
not get triggered).
Could you review this [~ste...@apache.org], if you spare some time?
> Disable retry of FailoverOnNetworkExceptionRetry in case of
> AccessControlException
> --
>
> Key: HADOOP-16580
> URL: https://issues.apache.org/jira/browse/HADOOP-16580
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16580.001.patch
>
>
> HADOOP-14982 handled the case where a SaslException is thrown. The issue
> still persists, since the exception that is thrown is an
> *AccessControlException* because user has no kerberos credentials.
> My suggestion is that we should add this case as well to
> {{FailoverOnNetworkExceptionRetry}}.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16580) Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException
[
https://issues.apache.org/jira/browse/HADOOP-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16580:
Status: Patch Available (was: In Progress)
> Disable retry of FailoverOnNetworkExceptionRetry in case of
> AccessControlException
> --
>
> Key: HADOOP-16580
> URL: https://issues.apache.org/jira/browse/HADOOP-16580
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16580.001.patch
>
>
> HADOOP-14982 handled the case where a SaslException is thrown. The issue
> still persists, since the exception that is thrown is an
> *AccessControlException* because user has no kerberos credentials.
> My suggestion is that we should add this case as well to
> {{FailoverOnNetworkExceptionRetry}}.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16580) Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException
[
https://issues.apache.org/jira/browse/HADOOP-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16580:
Attachment: HADOOP-16580.001.patch
> Disable retry of FailoverOnNetworkExceptionRetry in case of
> AccessControlException
> --
>
> Key: HADOOP-16580
> URL: https://issues.apache.org/jira/browse/HADOOP-16580
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16580.001.patch
>
>
> HADOOP-14982 handled the case where a SaslException is thrown. The issue
> still persists, since the exception that is thrown is an
> *AccessControlException* because user has no kerberos credentials.
> My suggestion is that we should add this case as well to
> {{FailoverOnNetworkExceptionRetry}}.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14982) Clients using FailoverOnNetworkExceptionRetry can go into a loop if they're used without authenticating with kerberos in HA env
[
https://issues.apache.org/jira/browse/HADOOP-14982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931434#comment-16931434
]
Adam Antal commented on HADOOP-14982:
-
Indeed, the issue in case of AccessControlException is still unresolved. Opened
HADOOP-16580 for the AccessControlException case.
> Clients using FailoverOnNetworkExceptionRetry can go into a loop if they're
> used without authenticating with kerberos in HA env
> ---
>
> Key: HADOOP-14982
> URL: https://issues.apache.org/jira/browse/HADOOP-14982
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.1.0, 2.10.0
>
> Attachments: HADOOP-14892-001.patch, HADOOP-14892-002.patch,
> HADOOP-14982-003.patch
>
>
> If HA is configured for the Resource Manager in a secure environment, using
> the mapred client goes into a loop if the user is not authenticated with
> Kerberos.
> {noformat}
> [root@pb6sec-1 ~]# mapred job -list
> 17/10/25 06:37:43 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
> to rm36
> 17/10/25 06:37:43 WARN ipc.Client: Exception encountered while connecting to
> the server : javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 17/10/25 06:37:43 INFO retry.RetryInvocationHandler: java.io.IOException:
> Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]; Host Details : local host is:
> "host_redacted/IP_redacted"; destination host is: "com.host2.redacted:8032; ,
> while invoking ApplicationClientProtocolPBClientImpl.getApplications over
> rm36 after 1 failover attempts. Trying to failover after sleeping for 160ms.
> 17/10/25 06:37:43 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
> to rm25
> 17/10/25 06:37:43 INFO retry.RetryInvocationHandler:
> java.net.ConnectException: Call From host_redacted/IP_redacted to
> com.host.redacted:8032 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused, while invoking
> ApplicationClientProtocolPBClientImpl.getApplications over rm25 after 2
> failover attempts. Trying to failover after sleeping for 582ms.
> 17/10/25 06:37:44 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
> to rm36
> 17/10/25 06:37:44 WARN ipc.Client: Exception encountered while connecting to
> the server : javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 17/10/25 06:37:44 INFO retry.RetryInvocationHandler: java.io.IOException:
> Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]; Host Details : local host is:
> "host_redacted/IP_redacted"; destination host is: "com.host2.redacted:8032; ,
> while invoking ApplicationClientProtocolPBClientImpl.getApplications over
> rm36 after 3 failover attempts. Trying to failover after sleeping for 977ms.
> 17/10/25 06:37:45 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
> to rm25
> 17/10/25 06:37:45 INFO retry.RetryInvocationHandler:
> java.net.ConnectException: Call From host_redacted/IP_redacted to
> com.host.redacted:8032 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused, while invoking
> ApplicationClientProtocolPBClientImpl.getApplications over rm25 after 4
> failover attempts. Trying to failover after sleeping for 1667ms.
> 17/10/25 06:37:46 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
> to rm36
> 17/10/25 06:37:46 WARN ipc.Client: Exception encountered while connecting to
> the server : javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 17/10/25 06:37:46 INFO retry.RetryInvocationHandler: java.io.IOException:
> Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]; Host Details : local host is:
> "host_redacted/IP_redacted"; destination host is:
[jira] [Created] (HADOOP-16580) Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException
Adam Antal created HADOOP-16580:
---
Summary: Disable retry of FailoverOnNetworkExceptionRetry in case
of AccessControlException
Key: HADOOP-16580
URL: https://issues.apache.org/jira/browse/HADOOP-16580
Project: Hadoop Common
Issue Type: Bug
Components: common
Affects Versions: 3.3.0
Reporter: Adam Antal
Assignee: Adam Antal
HADOOP-14982 handled the case where a SaslException is thrown. The issue still
persists, since the exception that is thrown is an *AccessControlException*
because user has no kerberos credentials.
My suggestion is that we should add this case as well to
{{FailoverOnNetworkExceptionRetry}}.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-16580) Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException
[
https://issues.apache.org/jira/browse/HADOOP-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-16580 started by Adam Antal.
---
> Disable retry of FailoverOnNetworkExceptionRetry in case of
> AccessControlException
> --
>
> Key: HADOOP-16580
> URL: https://issues.apache.org/jira/browse/HADOOP-16580
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
>
> HADOOP-14982 handled the case where a SaslException is thrown. The issue
> still persists, since the exception that is thrown is an
> *AccessControlException* because user has no kerberos credentials.
> My suggestion is that we should add this case as well to
> {{FailoverOnNetworkExceptionRetry}}.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925464#comment-16925464
]
Adam Antal commented on HADOOP-16503:
-
YARN-9784 has been committed (thanks to [~kmarton]).
[~kmarton] could you update the issue whether the fix in JDK8 does indeed
resolve the failure on JDK11 as well? Or do we need an additional patch to fix
it in JDK11?
> [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
> -
>
> Key: HADOOP-16503
> URL: https://issues.apache.org/jira/browse/HADOOP-16503
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: HADOOP-16503.001.patch
>
>
> Many of the tests in
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
> fails with the following error message running on JDK11:
> {noformat}
> [ERROR]
> testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
> Time elapsed: 0.204 s <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> YarnConfiguration cannot be returned by getRMNodes()
> getRMNodes() should return ConcurrentMap
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
> spies -
>- with doReturn|Throw() family of methods. More in javadocs for
> Mockito.spy() method.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
>...
> {noformat}
> This is due to the actual execution of the call, while we need to record only
> the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16512) [hadoop-tools] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922232#comment-16922232
]
Adam Antal commented on HADOOP-16512:
-
Hi [~pingsutw],
Strangely jenkins didn't pick up latest patch. Would you mind reuploading it
again?
> [hadoop-tools] Fix order of actual and expected expression in assert
> statements
> ---
>
> Key: HADOOP-16512
> URL: https://issues.apache.org/jira/browse/HADOOP-16512
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: kevin su
>Priority: Major
> Attachments: HADOOP-16512.001.patch
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up Jira on the fix for the hadoop-tools project.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16542) Update commons-beanutils version
[
https://issues.apache.org/jira/browse/HADOOP-16542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1694#comment-1694
]
Adam Antal commented on HADOOP-16542:
-
I agree with the above, +1 (non-binding) on patch v2 pending on jenkins.
> Update commons-beanutils version
>
>
> Key: HADOOP-16542
> URL: https://issues.apache.org/jira/browse/HADOOP-16542
> Project: Hadoop Common
> Issue Type: Task
>Affects Versions: 2.10.0, 3.3.0
>Reporter: Wei-Chiu Chuang
>Assignee: kevin su
>Priority: Major
> Labels: release-blocker
> Attachments: HADOOP-16542.001.patch, HADOOP-16542.002.patch
>
>
> [http://mail-archives.apache.org/mod_mbox/www-announce/201908.mbox/%3cc628798f-315d-4428-8cb1-4ed1ecc95...@apache.org%3e]
> {quote}
> CVE-2019-10086. Apache Commons Beanutils does not suppresses the class
> property in PropertyUtilsBean
> by default.
> Severity: Medium
> Vendor: The Apache Software Foundation
> Versions Affected: commons-beanutils-1.9.3 and earlier
> Description: A special BeanIntrospector class was added in version 1.9.2.
> This can be used to stop attackers from using the class property of
> Java objects to get access to the classloader.
> However this protection was not enabled by default.
> PropertyUtilsBean (and consequently BeanUtilsBean) now disallows class
> level property access by default, thus protecting against
> CVE-2014-0114.
> Mitigation: 1.X users should migrate to 1.9.4.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920832#comment-16920832
]
Adam Antal commented on HADOOP-16503:
-
It seems that this will be resolved in YARN-9784. Worth rerunning the tests
after that it will have been committed, but the root cause should be mitigated
by YARN-9784.
> [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
> -
>
> Key: HADOOP-16503
> URL: https://issues.apache.org/jira/browse/HADOOP-16503
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: HADOOP-16503.001.patch
>
>
> Many of the tests in
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
> fails with the following error message running on JDK11:
> {noformat}
> [ERROR]
> testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
> Time elapsed: 0.204 s <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> YarnConfiguration cannot be returned by getRMNodes()
> getRMNodes() should return ConcurrentMap
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
> spies -
>- with doReturn|Throw() family of methods. More in javadocs for
> Mockito.spy() method.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
>...
> {noformat}
> This is due to the actual execution of the call, while we need to record only
> the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906301#comment-16906301
]
Adam Antal commented on HADOOP-16503:
-
Verification failed, I still managed to see the exception after applying the
patch. Will look into this a bit further later.
> [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
> -
>
> Key: HADOOP-16503
> URL: https://issues.apache.org/jira/browse/HADOOP-16503
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16503.001.patch
>
>
> Many of the tests in
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
> fails with the following error message running on JDK11:
> {noformat}
> [ERROR]
> testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
> Time elapsed: 0.204 s <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> YarnConfiguration cannot be returned by getRMNodes()
> getRMNodes() should return ConcurrentMap
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
> spies -
>- with doReturn|Throw() family of methods. More in javadocs for
> Mockito.spy() method.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
>...
> {noformat}
> This is due to the actual execution of the call, while we need to record only
> the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16264) [JDK11] Track failing Hadoop unit tests
[ https://issues.apache.org/jira/browse/HADOOP-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906297#comment-16906297 ] Adam Antal commented on HADOOP-16264: - I also add my findings for the hadoop-yarn-project: [^test-run-openjdk11.0.4-macos10.14.5.txt]. > [JDK11] Track failing Hadoop unit tests > --- > > Key: HADOOP-16264 > URL: https://issues.apache.org/jira/browse/HADOOP-16264 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.1.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: test-run-openjdk11.0.4-macos10.14.5.txt, test-run1.tgz, > test-run2.log, test-run2.log.gz, test-run3-openjdk11.0.3u7-ubuntu19.04.log > > > Although there are still a lot of work before we could compile Hadoop with > JDK 11 (HADOOP-15338), it is possible to compile Hadoop with JDK 8 and run > (e.g. HDFS NN/DN,YARN NM/RM) on JDK 11 at this moment. > But after compiling branch-3.1.2 with JDK 8, I ran unit tests with JDK 11 and > there are a LOT of unit test failures (44 out of 96 maven projects contain at > least one unit test failures according to maven reactor summary). This may > well indicate some functionalities are actually broken on JDK 11. Some of > them already have a jira number. Some of them might have been fixed in 3.2.0. > Some of them might share the same root cause. > By definition, this jira should be part of HADOOP-15338. But the goal of this > one is just to keep track of unit test failures and (hopefully) resolve all > of them soon. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16264) [JDK11] Track failing Hadoop unit tests
[ https://issues.apache.org/jira/browse/HADOOP-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated HADOOP-16264: Attachment: test-run-openjdk11.0.4-macos10.14.5.txt > [JDK11] Track failing Hadoop unit tests > --- > > Key: HADOOP-16264 > URL: https://issues.apache.org/jira/browse/HADOOP-16264 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.1.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: test-run-openjdk11.0.4-macos10.14.5.txt, test-run1.tgz, > test-run2.log, test-run2.log.gz, test-run3-openjdk11.0.3u7-ubuntu19.04.log > > > Although there are still a lot of work before we could compile Hadoop with > JDK 11 (HADOOP-15338), it is possible to compile Hadoop with JDK 8 and run > (e.g. HDFS NN/DN,YARN NM/RM) on JDK 11 at this moment. > But after compiling branch-3.1.2 with JDK 8, I ran unit tests with JDK 11 and > there are a LOT of unit test failures (44 out of 96 maven projects contain at > least one unit test failures according to maven reactor summary). This may > well indicate some functionalities are actually broken on JDK 11. Some of > them already have a jira number. Some of them might have been fixed in 3.2.0. > Some of them might share the same root cause. > By definition, this jira should be part of HADOOP-15338. But the goal of this > one is just to keep track of unit test failures and (hopefully) resolve all > of them soon. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16264) [JDK11] Track failing Hadoop unit tests
[ https://issues.apache.org/jira/browse/HADOOP-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906029#comment-16906029 ] Adam Antal commented on HADOOP-16264: - Created a few Yarn-related test fixing jiras: HADOOP-16503, YARN-9741, YARN-9742, YARN-9743 as a starting point. I want to get through and fix them one by one. Will also post the results of the UTs on openjdk-11.0.4 soon focusing on the Yarn part. > [JDK11] Track failing Hadoop unit tests > --- > > Key: HADOOP-16264 > URL: https://issues.apache.org/jira/browse/HADOOP-16264 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.1.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: test-run1.tgz, test-run2.log, test-run2.log.gz, > test-run3-openjdk11.0.3u7-ubuntu19.04.log > > > Although there are still a lot of work before we could compile Hadoop with > JDK 11 (HADOOP-15338), it is possible to compile Hadoop with JDK 8 and run > (e.g. HDFS NN/DN,YARN NM/RM) on JDK 11 at this moment. > But after compiling branch-3.1.2 with JDK 8, I ran unit tests with JDK 11 and > there are a LOT of unit test failures (44 out of 96 maven projects contain at > least one unit test failures according to maven reactor summary). This may > well indicate some functionalities are actually broken on JDK 11. Some of > them already have a jira number. Some of them might have been fixed in 3.2.0. > Some of them might share the same root cause. > By definition, this jira should be part of HADOOP-15338. But the goal of this > one is just to keep track of unit test failures and (hopefully) resolve all > of them soon. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905995#comment-16905995
]
Adam Antal commented on HADOOP-16503:
-
For the verification of the patch one may use the following command:
{noformat}
mvn clean package -Dit.test=None
-Dtest=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
{noformat}
> [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
> -
>
> Key: HADOOP-16503
> URL: https://issues.apache.org/jira/browse/HADOOP-16503
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16503.001.patch
>
>
> Many of the tests in
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
> fails with the following error message running on JDK11:
> {noformat}
> [ERROR]
> testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
> Time elapsed: 0.204 s <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> YarnConfiguration cannot be returned by getRMNodes()
> getRMNodes() should return ConcurrentMap
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
> spies -
>- with doReturn|Throw() family of methods. More in javadocs for
> Mockito.spy() method.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
>...
> {noformat}
> This is due to the actual execution of the call, while we need to record only
> the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16503:
Attachment: HADOOP-16503.001.patch
> [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
> -
>
> Key: HADOOP-16503
> URL: https://issues.apache.org/jira/browse/HADOOP-16503
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16503.001.patch
>
>
> Many of the tests in
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
> fails with the following error message running on JDK11:
> {noformat}
> [ERROR]
> testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
> Time elapsed: 0.204 s <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> YarnConfiguration cannot be returned by getRMNodes()
> getRMNodes() should return ConcurrentMap
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
> spies -
>- with doReturn|Throw() family of methods. More in javadocs for
> Mockito.spy() method.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
>...
> {noformat}
> This is due to the actual execution of the call, while we need to record only
> the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905993#comment-16905993
]
Adam Antal commented on HADOOP-16503:
-
Indeed, sorry.
I remember uploading it and then closing the tab - the upload might have
failed, my bad. Uploaded: [^HADOOP-16503.001.patch].
> [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
> -
>
> Key: HADOOP-16503
> URL: https://issues.apache.org/jira/browse/HADOOP-16503
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16503.001.patch
>
>
> Many of the tests in
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
> fails with the following error message running on JDK11:
> {noformat}
> [ERROR]
> testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
> Time elapsed: 0.204 s <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> YarnConfiguration cannot be returned by getRMNodes()
> getRMNodes() should return ConcurrentMap
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
> spies -
>- with doReturn|Throw() family of methods. More in javadocs for
> Mockito.spy() method.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
>...
> {noformat}
> This is due to the actual execution of the call, while we need to record only
> the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16510) [hadoop-common] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal reassigned HADOOP-16510:
---
Assignee: Adam Antal
> [hadoop-common] Fix order of actual and expected expression in assert
> statements
>
>
> Key: HADOOP-16510
> URL: https://issues.apache.org/jira/browse/HADOOP-16510
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up jira for the hadoop-common project.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-16510) [hadoop-common] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-16510 started by Adam Antal.
---
> [hadoop-common] Fix order of actual and expected expression in assert
> statements
>
>
> Key: HADOOP-16510
> URL: https://issues.apache.org/jira/browse/HADOOP-16510
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up jira for the hadoop-common project.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12693) Many misusages of assertEquals(expected, actual)
[
https://issues.apache.org/jira/browse/HADOOP-12693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905364#comment-16905364
]
Adam Antal commented on HADOOP-12693:
-
[~templedf] Oh indeed, I thought those were already done.
As I checked some of the occurrences in the just-rough-approx.txt some of those
entries are valid, and still present in hadoop-common, hadoop-hdfs and
hadoop-tools projects. Filed HADOOP-16510, HADOOP-16511 and HADOOP-16512. Also
collected them under this jira as sub-tasks for better trackability. Hope you
don't mind.
> Many misusages of assertEquals(expected, actual)
>
>
> Key: HADOOP-12693
> URL: https://issues.apache.org/jira/browse/HADOOP-12693
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Akihiro Suda
>Priority: Trivial
> Attachments: just-rough-approx.txt
>
>
> The first arg of {{org.JUnit.Assert.assertEquals()}} should be an
> {{expected}} value, and the second one should be an {{actual}} value.
> {code}
> void assertEquals(T expected, T actual);
> {code}
> http://junit.org/apidocs/org/junit/Assert.html#assertEquals(java.lang.Object,
> java.lang.Object)
> However, there are so many violations in Hadoop, which can make a misleading
> message like this:
> {code}
> AssertionError: expected: but was:
> {code}.
> Please refer to {{just-rough-approx.txt}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16512) [hadoop-tools] Fix order of actual and expected expression in assert statements
Adam Antal created HADOOP-16512:
---
Summary: [hadoop-tools] Fix order of actual and expected
expression in assert statements
Key: HADOOP-16512
URL: https://issues.apache.org/jira/browse/HADOOP-16512
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 3.2.0
Reporter: Adam Antal
Fix order of actual and expected expression in assert statements which gives
misleading message when test case fails. Attached file has some of the places
where it is placed wrongly.
{code:java}
[ERROR]
testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
Time elapsed: 3.385 s <<< FAILURE!
java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
was:<0>
{code}
For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be used
for new test cases which avoids such mistakes.
This is a follow-up Jira on the fix for the hadoop-tools project.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16511) [hadoop-hdfs] Fix order of actual and expected expression in assert statements
Adam Antal created HADOOP-16511:
---
Summary: [hadoop-hdfs] Fix order of actual and expected expression
in assert statements
Key: HADOOP-16511
URL: https://issues.apache.org/jira/browse/HADOOP-16511
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 3.2.0
Reporter: Adam Antal
Fix order of actual and expected expression in assert statements which gives
misleading message when test case fails. Attached file has some of the places
where it is placed wrongly.
{code:java}
[ERROR]
testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
Time elapsed: 3.385 s <<< FAILURE!
java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
was:<0>
{code}
For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be used
for new test cases which avoids such mistakes.
This is a follow-up jira for the hadoop-hdfs project.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16510) [hadoop-common] Fix order of actual and expected expression in assert statements
Adam Antal created HADOOP-16510:
---
Summary: [hadoop-common] Fix order of actual and expected
expression in assert statements
Key: HADOOP-16510
URL: https://issues.apache.org/jira/browse/HADOOP-16510
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 3.2.0
Reporter: Adam Antal
Fix order of actual and expected expression in assert statements which gives
misleading message when test case fails. Attached file has some of the places
where it is placed wrongly.
{code:java}
[ERROR]
testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
Time elapsed: 3.385 s <<< FAILURE!
java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
was:<0>
{code}
For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be used
for new test cases which avoids such mistakes.
This is a follow-up jira for the hadoop-common project.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-16509) [hadoop-mapreduce-project] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal moved MAPREDUCE-7197 to HADOOP-16509:
Fix Version/s: (was: 3.3.0)
3.3.0
Affects Version/s: (was: 3.2.0)
3.2.0
Target Version/s: (was: 3.3.0)
Key: HADOOP-16509 (was: MAPREDUCE-7197)
Project: Hadoop Common (was: Hadoop Map/Reduce)
> [hadoop-mapreduce-project] Fix order of actual and expected expression in
> assert statements
> ---
>
> Key: HADOOP-16509
> URL: https://issues.apache.org/jira/browse/HADOOP-16509
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: MAPREDUCE-7197.001.patch, MAPREDUCE-7197.002.patch,
> MAPREDUCE-7197.003.patch, MAPREDUCE-7197.004.patch, MAPREDUCE-7197.004.patch,
> MAPREDUCE-7197.005.patch, MAPREDUCE-7197.006.patch,
> old-approx-from-HADOOP-12693
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up Jira on the fix for the MR project.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16509) [hadoop-mapreduce-project] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16509:
Issue Type: Sub-task (was: Improvement)
Parent: HADOOP-12693
> [hadoop-mapreduce-project] Fix order of actual and expected expression in
> assert statements
> ---
>
> Key: HADOOP-16509
> URL: https://issues.apache.org/jira/browse/HADOOP-16509
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: MAPREDUCE-7197.001.patch, MAPREDUCE-7197.002.patch,
> MAPREDUCE-7197.003.patch, MAPREDUCE-7197.004.patch, MAPREDUCE-7197.004.patch,
> MAPREDUCE-7197.005.patch, MAPREDUCE-7197.006.patch,
> old-approx-from-HADOOP-12693
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up Jira on the fix for the MR project.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16508) [hadoop-yarn-project] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16508:
Summary: [hadoop-yarn-project] Fix order of actual and expected expression
in assert statements (was: Fix order of actual and expected expression in
assert statements)
> [hadoop-yarn-project] Fix order of actual and expected expression in assert
> statements
> --
>
> Key: HADOOP-16508
> URL: https://issues.apache.org/jira/browse/HADOOP-16508
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9470-001.patch, YARN-9470-002.patch,
> YARN-9470-003.patch, YARN-9470-004.patch, YARN-9470-005.patch, assertEquals
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16508) Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16508:
Issue Type: Sub-task (was: Improvement)
Parent: HADOOP-12693
> Fix order of actual and expected expression in assert statements
>
>
> Key: HADOOP-16508
> URL: https://issues.apache.org/jira/browse/HADOOP-16508
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9470-001.patch, YARN-9470-002.patch,
> YARN-9470-003.patch, YARN-9470-004.patch, YARN-9470-005.patch, assertEquals
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-16508) Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HADOOP-16508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal moved YARN-9470 to HADOOP-16508:
---
Fix Version/s: (was: 3.3.0)
3.3.0
Affects Version/s: (was: 3.2.0)
3.2.0
Component/s: (was: yarn)
Key: HADOOP-16508 (was: YARN-9470)
Project: Hadoop Common (was: Hadoop YARN)
> Fix order of actual and expected expression in assert statements
>
>
> Key: HADOOP-16508
> URL: https://issues.apache.org/jira/browse/HADOOP-16508
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9470-001.patch, YARN-9470-002.patch,
> YARN-9470-003.patch, YARN-9470-004.patch, YARN-9470-005.patch, assertEquals
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16503:
Status: Patch Available (was: Open)
> [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
> -
>
> Key: HADOOP-16503
> URL: https://issues.apache.org/jira/browse/HADOOP-16503
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
>
> Many of the tests in
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
> fails with the following error message running on JDK11:
> {noformat}
> [ERROR]
> testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
> Time elapsed: 0.204 s <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> YarnConfiguration cannot be returned by getRMNodes()
> getRMNodes() should return ConcurrentMap
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
> spies -
>- with doReturn|Throw() family of methods. More in javadocs for
> Mockito.spy() method.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
>...
> {noformat}
> This is due to the actual execution of the call, while we need to record only
> the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905318#comment-16905318
]
Adam Antal edited comment on HADOOP-16503 at 8/12/19 3:40 PM:
--
Uploaded patchset v1.
Changes:
- prefering doReturn(...).when(...).method(..) over when(...).thenReturn()
statements to mitigate the actual calls on the spies and mocks
- moved the mocking of the method {{getScheduler}} of the {{cs}} spy object to
a few lines above. An
org.mockito.exceptions.misusing.UnfinishedStubbingException is thrown
otherwise. I am not sure about the root cause, because in this form mocking
should have been finished, presumably there can be something inside
{{cs.start()}} (starting some threads in the background) that can be a problem.
Pls review.
was (Author: adam.antal):
Uploaded patchset v1.
Changes:
- prefering doReturn(...).when(...).method(..) over when(...).thenReturn()
statements to mitigate the actual calls on the spies and mocks
- moved the mocking of the method {{getScheduler}} of the {{cs}} spy object to
a few lines above. An
org.mockito.exceptions.misusing.UnfinishedStubbingException is thrown
otherwise. I am not sure about the root cause, because this form is finished in
this way, but there can be something inside {{cs.start()}} that can be a
problem.
Pls review.
> [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
> -
>
> Key: HADOOP-16503
> URL: https://issues.apache.org/jira/browse/HADOOP-16503
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
>
> Many of the tests in
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
> fails with the following error message running on JDK11:
> {noformat}
> [ERROR]
> testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
> Time elapsed: 0.204 s <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> YarnConfiguration cannot be returned by getRMNodes()
> getRMNodes() should return ConcurrentMap
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
> spies -
>- with doReturn|Throw() family of methods. More in javadocs for
> Mockito.spy() method.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
>...
> {noformat}
> This is due to the actual execution of the call, while we need to record only
> the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905318#comment-16905318
]
Adam Antal commented on HADOOP-16503:
-
Uploaded patchset v1.
Changes:
- prefering doReturn(...).when(...).method(..) over when(...).thenReturn()
statements to mitigate the actual calls on the spies and mocks
- moved the mocking of the method {{getScheduler}} of the {{cs}} spy object to
a few lines above. An
org.mockito.exceptions.misusing.UnfinishedStubbingException is thrown
otherwise. I am not sure about the root cause, because this form is finished in
this way, but there can be something inside {{cs.start()}} that can be a
problem.
Pls review.
> [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
> -
>
> Key: HADOOP-16503
> URL: https://issues.apache.org/jira/browse/HADOOP-16503
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
>
> Many of the tests in
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
> fails with the following error message running on JDK11:
> {noformat}
> [ERROR]
> testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
> Time elapsed: 0.204 s <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> YarnConfiguration cannot be returned by getRMNodes()
> getRMNodes() should return ConcurrentMap
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
> spies -
>- with doReturn|Throw() family of methods. More in javadocs for
> Mockito.spy() method.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
>...
> {noformat}
> This is due to the actual execution of the call, while we need to record only
> the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12693) Many misusages of assertEquals(expected, actual)
[
https://issues.apache.org/jira/browse/HADOOP-12693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905132#comment-16905132
]
Adam Antal commented on HADOOP-12693:
-
MAPREDUCE-7197 got committed. Is there anything left for this issue?
I suggest to close this. Any objections?
> Many misusages of assertEquals(expected, actual)
>
>
> Key: HADOOP-12693
> URL: https://issues.apache.org/jira/browse/HADOOP-12693
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Akihiro Suda
>Priority: Trivial
> Attachments: just-rough-approx.txt
>
>
> The first arg of {{org.JUnit.Assert.assertEquals()}} should be an
> {{expected}} value, and the second one should be an {{actual}} value.
> {code}
> void assertEquals(T expected, T actual);
> {code}
> http://junit.org/apidocs/org/junit/Assert.html#assertEquals(java.lang.Object,
> java.lang.Object)
> However, there are so many violations in Hadoop, which can make a misleading
> message like this:
> {code}
> AssertionError: expected: but was:
> {code}.
> Please refer to {{just-rough-approx.txt}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
Adam Antal created HADOOP-16503:
---
Summary: [JDK11] TestLeafQueue tests are failing due to
WrongTypeOfReturnValue
Key: HADOOP-16503
URL: https://issues.apache.org/jira/browse/HADOOP-16503
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 3.2.0
Reporter: Adam Antal
Assignee: Adam Antal
Many of the tests in
{{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
fails with the following error message running on JDK11:
{noformat}
[ERROR]
testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
Time elapsed: 0.204 s <<< ERROR!
org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
YarnConfiguration cannot be returned by getRMNodes()
getRMNodes() should return ConcurrentMap
***
If you're unsure why you're getting above error read on.
Due to the nature of the syntax above problem might occur because:
1. This exception *might* occur in wrongly written multi-threaded tests.
Please refer to Mockito FAQ on limitations of concurrency testing.
2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
spies -
- with doReturn|Throw() family of methods. More in javadocs for
Mockito.spy() method.
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
...
{noformat}
This is due to the actual execution of the call, while we need to record only
the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16496) Apply HDDS-1870 (ConcurrentModification at PrometheusMetricsSink) to Hadoop common
[
https://issues.apache.org/jira/browse/HADOOP-16496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16496:
Description: Hadoop common side of HDDS-1870. (was: {color:red}colored
text{color}Hadoop common side of HDDS-1870.)
> Apply HDDS-1870 (ConcurrentModification at PrometheusMetricsSink) to Hadoop
> common
> --
>
> Key: HADOOP-16496
> URL: https://issues.apache.org/jira/browse/HADOOP-16496
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Akira Ajisaka
>Priority: Major
>
> Hadoop common side of HDDS-1870.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16496) Apply HDDS-1870 (ConcurrentModification at PrometheusMetricsSink) to Hadoop common
[
https://issues.apache.org/jira/browse/HADOOP-16496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16496:
Description: {color:red}colored text{color}Hadoop common side of HDDS-1870.
(was: Hadoop common side of HDDS-1870.)
> Apply HDDS-1870 (ConcurrentModification at PrometheusMetricsSink) to Hadoop
> common
> --
>
> Key: HADOOP-16496
> URL: https://issues.apache.org/jira/browse/HADOOP-16496
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Akira Ajisaka
>Priority: Major
>
> {color:red}colored text{color}Hadoop common side of HDDS-1870.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16377) Moving logging APIs over to slf4j
[
https://issues.apache.org/jira/browse/HADOOP-16377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902812#comment-16902812
]
Adam Antal commented on HADOOP-16377:
-
Since Submarine will be a separate Apache project soon, and lots of the Jenkins
slaves are broken when running tests against Submarine I'd suggest to create a
separate ticket in the Submarine project for this issue. That would also enable
us to have a clear jenkins result excluding some of those nasty error messages.
> Moving logging APIs over to slf4j
> -
>
> Key: HADOOP-16377
> URL: https://issues.apache.org/jira/browse/HADOOP-16377
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: HADOOP-16357-002.patch, HADOOP-16377-001.patch,
> HADOOP-16377-003.patch, HADOOP-16377-004.patch, HADOOP-16377-005.patch,
> HADOOP-16377-006.patch, HADOOP-16377-007.patch
>
>
> As of today, there are still 50 references to log4j1
> {code}
> $ grep -r "import org.apache.commons.logging.Log;" . |wc - l
> 50
> {code}
> To achieve the goal of HADOOP-12956/HADOOP-16206, we should invest time to
> move them to slf4j
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16264) [JDK11] Track failing Hadoop unit tests
[ https://issues.apache.org/jira/browse/HADOOP-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902791#comment-16902791 ] Adam Antal commented on HADOOP-16264: - Hi [~smeng], Do you have an up-to-date list of the still failing tests in JDK11? > [JDK11] Track failing Hadoop unit tests > --- > > Key: HADOOP-16264 > URL: https://issues.apache.org/jira/browse/HADOOP-16264 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.1.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: test-run1.tgz, test-run2.log, test-run2.log.gz, > test-run3-openjdk11.0.3u7-ubuntu19.04.log > > > Although there are still a lot of work before we could compile Hadoop with > JDK 11 (HADOOP-15338), it is possible to compile Hadoop with JDK 8 and run > (e.g. HDFS NN/DN,YARN NM/RM) on JDK 11 at this moment. > But after compiling branch-3.1.2 with JDK 8, I ran unit tests with JDK 11 and > there are a LOT of unit test failures (44 out of 96 maven projects contain at > least one unit test failures according to maven reactor summary). This may > well indicate some functionalities are actually broken on JDK 11. Some of > them already have a jira number. Some of them might have been fixed in 3.2.0. > Some of them might share the same root cause. > By definition, this jira should be part of HADOOP-15338. But the goal of this > one is just to keep track of unit test failures and (hopefully) resolve all > of them soon. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16398) Exports Hadoop metrics to Prometheus
[
https://issues.apache.org/jira/browse/HADOOP-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888720#comment-16888720
]
Adam Antal commented on HADOOP-16398:
-
Hi,
It seems a bit strange to me that HDFS depends on a HDDS config. Wouldn't it
make more sense to add this "hack" on HDDS side? I specifically mean
{{HttpServer2#addPrometheusServlet}}
Otherwise the patch LGTM (non-binding).
> Exports Hadoop metrics to Prometheus
>
>
> Key: HADOOP-16398
> URL: https://issues.apache.org/jira/browse/HADOOP-16398
> Project: Hadoop Common
> Issue Type: New Feature
> Components: metrics
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-16398.001.patch
>
>
> Hadoop common side of HDDS-846. HDDS already have its own
> PrometheusMetricsSink, so we can reuse the implementation.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882108#comment-16882108
]
Adam Antal commented on HADOOP-15691:
-
Hi [~ste...@apache.org], is there any update on this?
Would be happy to see this committed for YARN-9607 and YARN-9525.
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch, HADOOP-15691-004.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of a filesystem through both the
> FileSystem and FileContext APIs
> This is needed for
> * HADOOP-14707: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868449#comment-16868449
]
Adam Antal commented on HADOOP-15691:
-
Reviewed and had some questions commented on the PR.
Great job so far [~ste...@apache.org].
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch, HADOOP-15691-004.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of a filesystem through both the
> FileSystem and FileContext APIs
> This is needed for
> * HADOOP-14707: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15914) hadoop jar command has no help argument
[
https://issues.apache.org/jira/browse/HADOOP-15914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866326#comment-16866326
]
Adam Antal commented on HADOOP-15914:
-
Thank you for the commit [~jojochuang], and [~templedf] for the review.
> hadoop jar command has no help argument
> ---
>
> Key: HADOOP-15914
> URL: https://issues.apache.org/jira/browse/HADOOP-15914
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HADOOP-15914.000.patch
>
>
> {{hadoop jar --help}} and {{hadoop jar help}} commands show outputs like this:
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> JAR does not exist or is not a normal file: /root/--help
> {noformat}
> Only if called with no arguments: {{hadoop jar}} we get the usage text, but
> even in that case we get:
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> RunJar jarFile [mainClass] args...
> {noformat}
> Where RunJar is wrapped by the hadoop script (so it should not be displayed).
> {{hadoop --help}} displays the following:
> {noformat}
> jar run a jar file. NOTE: please use "yarn jar" to launch YARN
> applications, not this command.
> {noformat}
> which is fine, but {{CommandsManual.md}} tells a bit more information about
> the usage of this command:
> {noformat}
> Usage: hadoop jar [mainClass] args...
> {noformat}
> My suggestion is to add a {{--help}} option to the {{hadoop jar}} command
> that would display this message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16263) Update BUILDING.txt with macOS native build instructions
[ https://issues.apache.org/jira/browse/HADOOP-16263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844572#comment-16844572 ] Adam Antal commented on HADOOP-16263: - I followed the steps and I successfully compiled native code on macOS. One minor comment though: brew is raising errors because it could not find zlib1g-dev package. You wrote that the rest are optional in that step, so I'd recommend letting it out completely. If it's not necessary, then we don't need it - it would just cause errors, like for me. Other than that, it's +1 (non-binding). > Update BUILDING.txt with macOS native build instructions > > > Key: HADOOP-16263 > URL: https://issues.apache.org/jira/browse/HADOOP-16263 > Project: Hadoop Common > Issue Type: Task >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Minor > Attachments: HADOOP-16263.001.patch, HADOOP-16263.002.patch > > > I recently tried to compile Hadoop native on a Mac and found a few catches, > involving fixing some YARN native compiling issues (YARN-8622, YARN-9487). > Also, need to specify OpenSSL (brewed) header include dir when building > native with maven on a Mac. Should update BUILDING.txt for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16263) Update BUILDING.txt with macOS native build instructions
[ https://issues.apache.org/jira/browse/HADOOP-16263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840268#comment-16840268 ] Adam Antal commented on HADOOP-16263: - Sorry didn't have the chance to try this yet, but I hope I can test it this week. > Update BUILDING.txt with macOS native build instructions > > > Key: HADOOP-16263 > URL: https://issues.apache.org/jira/browse/HADOOP-16263 > Project: Hadoop Common > Issue Type: Task >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Minor > Attachments: HADOOP-16263.001.patch, HADOOP-16263.002.patch > > > I recently tried to compile Hadoop native on a Mac and found a few catches, > involving fixing some YARN native compiling issues (YARN-8622, YARN-9487). > Also, need to specify OpenSSL (brewed) header include dir when building > native with maven on a Mac. Should update BUILDING.txt for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16263) Update BUILDING.txt with macOS native build instructions
[ https://issues.apache.org/jira/browse/HADOOP-16263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825846#comment-16825846 ] Adam Antal commented on HADOOP-16263: - Thanks for the patch [~smeng]! I will do the same steps ensuring nothing is accidentally missing. I'd rather emphasise that using homebrew is not mandatory. You can manually install all the install dependencies on your own, or you can use a tool that helps you to do it like homebrew, but even in that case homebrew is not exclusive. Also YARN-8622 and YARN-9487 got committed to branch-3.2, so it will be in the next maintenance release on the 3.2 line. I would just simply say that 3.2.0 will not compile, but 3.2.1 does. Thus one doesn't have to bother with the backport. > Update BUILDING.txt with macOS native build instructions > > > Key: HADOOP-16263 > URL: https://issues.apache.org/jira/browse/HADOOP-16263 > Project: Hadoop Common > Issue Type: Task >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Minor > Attachments: HADOOP-16263.001.patch > > > I recently tried to compile Hadoop native on a Mac and found a few catches, > involving fixing some YARN native compiling issues (YARN-8622, YARN-9487). > Also, need to specify OpenSSL (brewed) header include dir when building > native with maven on a Mac. Should update BUILDING.txt for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16082) FsShell ls: Add option -i to print inode id
[ https://issues.apache.org/jira/browse/HADOOP-16082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824083#comment-16824083 ] Adam Antal commented on HADOOP-16082: - I might not get the full context of this, but as HdfsLocatedFileStatus is intended for private use so its inode-id. The only occasion when inode-id is communicating with the clients is when we ls into "/.reserved/.inodes/...". I was wondering whether the reserved virtual uri would solve the problem - something like "/.reserved/.listId/path/to/hdfs/dir" would be good for this? > FsShell ls: Add option -i to print inode id > --- > > Key: HADOOP-16082 > URL: https://issues.apache.org/jira/browse/HADOOP-16082 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.2.0, 3.1.1 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HADOOP-16082.001.patch > > > When debugging the FSImage corruption issue, I often need to know a file's or > directory's inode id. At this moment, the only way to do that is to use OIV > tool to dump the FSImage and look up the filename, which is very inefficient. > Here I propose adding option "-i" in FsShell that prints files' or > directories' inode id. > h2. Implementation > h3. For hdfs:// (HDFS) > fileId exists in HdfsLocatedFileStatus, which is already returned to > hdfs-client. We just need to print it in Ls#processPath(). > h3. For file:// (Local FS) > h4. Linux > Use java.nio. > h4. Windows > Windows has the concept of "File ID" which is similar to inode id. It is > unique in NTFS and ReFS. > h3. For other FS > The fileId entry will be "0" in FileStatus if it is not set. We could either > ignore or throw an exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12693) Many misusages of assertEquals(expected, actual)
[
https://issues.apache.org/jira/browse/HADOOP-12693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821137#comment-16821137
]
Adam Antal commented on HADOOP-12693:
-
Filed MAPREDUCE-7197 for creating a follow-up for the MR project.
> Many misusages of assertEquals(expected, actual)
>
>
> Key: HADOOP-12693
> URL: https://issues.apache.org/jira/browse/HADOOP-12693
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Akihiro Suda
>Priority: Trivial
> Attachments: just-rough-approx.txt
>
>
> The first arg of {{org.JUnit.Assert.assertEquals()}} should be an
> {{expected}} value, and the second one should be an {{actual}} value.
> {code}
> void assertEquals(T expected, T actual);
> {code}
> http://junit.org/apidocs/org/junit/Assert.html#assertEquals(java.lang.Object,
> java.lang.Object)
> However, there are so many violations in Hadoop, which can make a misleading
> message like this:
> {code}
> AssertionError: expected: but was:
> {code}.
> Please refer to {{just-rough-approx.txt}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16264) [JDK11] Track failing Hadoop unit tests
[ https://issues.apache.org/jira/browse/HADOOP-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821117#comment-16821117 ] Adam Antal commented on HADOOP-16264: - Thanks for filing this [~smeng]. Since HADOOP-15338 is an umbrella jira, it would make sense to assign it a subtask of the uber-Jira - but still collecting the unit test failures here. Also, can you upload the maven reactor summary and explicitly naming the failing unit test on JDK 11? That would be a great starting point. > [JDK11] Track failing Hadoop unit tests > --- > > Key: HADOOP-16264 > URL: https://issues.apache.org/jira/browse/HADOOP-16264 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.1.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > Although there are still a lot of work before we could compile Hadoop with > JDK 11 (HADOOP-15338), it is possible to compile Hadoop with JDK 8 and run > (e.g. HDFS NN/DN,YARN NM/RM) on JDK 11 at this moment. > But after compiling branch-3.1.2 with JDK 8, I ran unit tests with JDK 11 and > there are a LOT of unit test failures (44 out of 96 maven projects contain at > least one unit test failures according to maven reactor summary). This may > well indicate some functionalities are actually broken on JDK 11. Some of > them already have a jira number. Some of them might have been fixed in 3.2.0. > Some of them might share the same root cause. > By definition, this jira should be part of HADOOP-15338. But the goal of this > one is just to keep track of unit test failures and (hopefully) resolve all > of them soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16238) Add the possbility to set SO_REUSEADDR in IPC Server Listener
[
https://issues.apache.org/jira/browse/HADOOP-16238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818186#comment-16818186
]
Adam Antal commented on HADOOP-16238:
-
Thanks [~pbacsko], I trust your findings, +1 (non-binding).
Also, I like the idea that we should disable it if not enabled explicitly. If
we don't break this anyways, I'm happy.
> Add the possbility to set SO_REUSEADDR in IPC Server Listener
> -
>
> Key: HADOOP-16238
> URL: https://issues.apache.org/jira/browse/HADOOP-16238
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Minor
> Attachments: HADOOP-16238-001.patch, HADOOP-16238-002.patch,
> HADOOP-16238-003.patch, HADOOP-16238-004.patch
>
>
> Currently we can't enable SO_REUSEADDR in the IPC Server. In some
> circumstances, this would be desirable, see explanation here:
> [https://developer.ibm.com/tutorials/l-sockpit/#pitfall-3-address-in-use-error-eaddrinuse-]
> Rarely it also causes problems in a test case
> {{TestMiniMRClientCluster.testRestart}}:
> {noformat}
> 2019-04-04 11:21:31,896 INFO [main] service.AbstractService
> (AbstractService.java:noteFailure(273)) - Service
> org.apache.hadoop.yarn.server.resourcemanager.AdminService failed in state
> STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.net.BindException: Problem binding to [test-host:35491]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.net.BindException: Problem binding to [test-host:35491]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:138)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
> at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:178)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1244)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:355)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.access$300(MiniYARNCluster.java:127)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:493)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:312)
> at
> org.apache.hadoop.mapreduce.v2.MiniMRYarnCluster.serviceStart(MiniMRYarnCluster.java:210)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapred.MiniMRYarnClusterAdapter.restart(MiniMRYarnClusterAdapter.java:73)
> at
> org.apache.hadoop.mapred.TestMiniMRClientCluster.testRestart(TestMiniMRClientCluster.java:114)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62){noformat}
>
> At least for testing, having this socket option enabled is benefical. We
> could enable this with a new property like {{ipc.server.reuseaddr}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16238) Add the possbility to set SO_REUSEADDR in IPC Server Listener
[
https://issues.apache.org/jira/browse/HADOOP-16238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815247#comment-16815247
]
Adam Antal commented on HADOOP-16238:
-
Thanks for the patch, [~pbacsko].
Do you sure that this option is always disabled? Because after your patch is
will be, if not enabled explicitly. That could alter the existing behaviour.
> Add the possbility to set SO_REUSEADDR in IPC Server Listener
> -
>
> Key: HADOOP-16238
> URL: https://issues.apache.org/jira/browse/HADOOP-16238
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Minor
> Attachments: HADOOP-16238-001.patch, HADOOP-16238-002.patch,
> HADOOP-16238-003.patch, HADOOP-16238-004.patch
>
>
> Currently we can't enable SO_REUSEADDR in the IPC Server. In some
> circumstances, this would be desirable, see explanation here:
> [https://developer.ibm.com/tutorials/l-sockpit/#pitfall-3-address-in-use-error-eaddrinuse-]
> Rarely it also causes problems in a test case
> {{TestMiniMRClientCluster.testRestart}}:
> {noformat}
> 2019-04-04 11:21:31,896 INFO [main] service.AbstractService
> (AbstractService.java:noteFailure(273)) - Service
> org.apache.hadoop.yarn.server.resourcemanager.AdminService failed in state
> STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.net.BindException: Problem binding to [test-host:35491]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.net.BindException: Problem binding to [test-host:35491]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:138)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
> at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.startServer(AdminService.java:178)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.serviceStart(AdminService.java:165)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1244)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:355)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.access$300(MiniYARNCluster.java:127)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:493)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:312)
> at
> org.apache.hadoop.mapreduce.v2.MiniMRYarnCluster.serviceStart(MiniMRYarnCluster.java:210)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapred.MiniMRYarnClusterAdapter.restart(MiniMRYarnClusterAdapter.java:73)
> at
> org.apache.hadoop.mapred.TestMiniMRClientCluster.testRestart(TestMiniMRClientCluster.java:114)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62){noformat}
>
> At least for testing, having this socket option enabled is benefical. We
> could enable this with a new property like {{ipc.server.reuseaddr}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16168) mvn clean site is not compiling in trunk
[
https://issues.apache.org/jira/browse/HADOOP-16168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808528#comment-16808528
]
Adam Antal commented on HADOOP-16168:
-
{{mvn clean site}} compiles perfectly. The issue can be closed.
> mvn clean site is not compiling in trunk
>
>
> Key: HADOOP-16168
> URL: https://issues.apache.org/jira/browse/HADOOP-16168
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.2.1
>Reporter: Adam Antal
>Assignee: Fengnan Li
>Priority: Blocker
>
> This is a follow-up Jira for HDFS-14118.
> {{mvn clean site}} is not compiling on trunk with the following error message:
> {noformat}
> [INFO] -
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /Users/adamantal/git/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConfiguredFailoverProxyProvider.java:[23,29]
> cannot find symbol
> symbol: class MockDomainNameResolver
> location: package org.apache.hadoop.net
> [ERROR]
> /Users/adamantal/git/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConfiguredFailoverProxyProvider.java:[149,11]
> cannot find symbol
> symbol: variable MockDomainNameResolver
> location: class
> org.apache.hadoop.hdfs.server.namenode.ha.TestConfiguredFailoverProxyProvider
> [ERROR]
> /Users/adamantal/git/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConfiguredFailoverProxyProvider.java:[150,11]
> cannot find symbol
> symbol: variable MockDomainNameResolver
> location: class
> org.apache.hadoop.hdfs.server.namenode.ha.TestConfiguredFailoverProxyProvider
> [ERROR]
> /Users/adamantal/git/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConfiguredFailoverProxyProvider.java:[162,9]
> cannot find symbol
> symbol: class MockDomainNameResolver
> location: class
> org.apache.hadoop.hdfs.server.namenode.ha.TestConfiguredFailoverProxyProvider
> [ERROR]
> /Users/adamantal/git/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConfiguredFailoverProxyProvider.java:[261,9]
> cannot find symbol
> symbol: variable MockDomainNameResolver
> location: class
> org.apache.hadoop.hdfs.server.namenode.ha.TestConfiguredFailoverProxyProvider
> [ERROR]
> /Users/adamantal/git/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConfiguredFailoverProxyProvider.java:[263,9]
> cannot find symbol
> symbol: variable MockDomainNameResolver
> location: class
> org.apache.hadoop.hdfs.server.namenode.ha.TestConfiguredFailoverProxyProvider
> [ERROR]
> /Users/adamantal/git/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConfiguredFailoverProxyProvider.java:[288,19]
> cannot find symbol
> symbol: variable MockDomainNameResolver
> location: class
> org.apache.hadoop.hdfs.server.namenode.ha.TestConfiguredFailoverProxyProvider
> [ERROR]
> /Users/adamantal/git/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConfiguredFailoverProxyProvider.java:[292,19]
> cannot find symbol
> symbol: variable MockDomainNameResolver
> location: class
> org.apache.hadoop.hdfs.server.namenode.ha.TestConfiguredFailoverProxyProvider
> {noformat}
> {{MockDomainNameResolver}} is in
> {{hadoop-common-project/hadoop-common/src/test}} while
> {{TestConfiguredFailoverProxyProvider}} is in
> {{hadoop-hdfs-project/hadoop-hdfs-client/src/test}}.
> Though we have the following dependency:
> {noformat}
>
> org.apache.hadoop
> hadoop-common
> test
> test-jar
>
> {noformat}
> probably that's not enough.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16124) Extend documentation in testing.md about endpoint constants
[ https://issues.apache.org/jira/browse/HADOOP-16124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799082#comment-16799082 ] Adam Antal commented on HADOOP-16124: - Thanks for the commit, [~ste...@apache.org]! > Extend documentation in testing.md about endpoint constants > --- > > Key: HADOOP-16124 > URL: https://issues.apache.org/jira/browse/HADOOP-16124 > Project: Hadoop Common > Issue Type: Sub-task > Components: hadoop-aws >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Trivial > Fix For: 3.3.0, 3.2.1 > > Attachments: HADOOP-16124.001.patch > > > Since HADOOP-14190 we had shortcuts for endpoints in the core-site.xml in > hadoop-aws. This is useful to know when someone come across testing in > hadoop-aws, so I suggest to add this little addition to the testing.md. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
