[jira] [Comment Edited] (HADOOP-12082) Support multiple authentication schemes via AuthenticationFilter
[ https://issues.apache.org/jira/browse/HADOOP-12082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570578#comment-15570578 ] Benoy Antony edited comment on HADOOP-12082 at 10/13/16 5:19 AM: - Looks good. Just a suggestion for documentation. I think , its better to change the filter-name in the example to "authFilter" instead of "kerberosFilter" . was (Author: benoyantony): Looks good. Just a suggestion for documentation. I think , its more appropriate to change the filter-name in the example to "authFilter" instead "kerberosFilter" . > Support multiple authentication schemes via AuthenticationFilter > > > Key: HADOOP-12082 > URL: https://issues.apache.org/jira/browse/HADOOP-12082 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.6.0 >Reporter: Hrishikesh Gadre >Assignee: Hrishikesh Gadre > Attachments: HADOOP-12082-001.patch, HADOOP-12082.patch, > hadoop-ldap-auth-v2.patch, hadoop-ldap-auth-v3.patch, > hadoop-ldap-auth-v4.patch, hadoop-ldap-auth-v5.patch, > hadoop-ldap-auth-v6.patch, hadoop-ldap.patch, > multi-scheme-auth-support-poc.patch > > > The requirement is to support LDAP based authentication scheme via Hadoop > AuthenticationFilter. HADOOP-9054 added a support to plug-in custom > authentication scheme (in addition to Kerberos) via > AltKerberosAuthenticationHandler class. But it is based on selecting the > authentication mechanism based on User-Agent HTTP header which does not > conform to HTTP protocol semantics. > As per [RFC-2616|http://www.w3.org/Protocols/rfc2616/rfc2616.html] > - HTTP protocol provides a simple challenge-response authentication mechanism > that can be used by a server to challenge a client request and by a client to > provide the necessary authentication information. > - This mechanism is initiated by server sending the 401 (Authenticate) > response with ‘WWW-Authenticate’ header which includes at least one challenge > that indicates the authentication scheme(s) and parameters applicable to the > Request-URI. > - In case server supports multiple authentication schemes, it may return > multiple challenges with a 401 (Authenticate) response, and each challenge > may use a different auth-scheme. > - A user agent MUST choose to use the strongest auth-scheme it understands > and request credentials from the user based upon that challenge. > The existing Hadoop authentication filter implementation supports Kerberos > authentication scheme and uses ‘Negotiate’ as the challenge as part of > ‘WWW-Authenticate’ response header. As per the following documentation, > ‘Negotiate’ challenge scheme is only applicable to Kerberos (and Windows > NTLM) authentication schemes. > [SPNEGO-based Kerberos and NTLM HTTP > Authentication|http://tools.ietf.org/html/rfc4559] > [Understanding HTTP > Authentication|https://msdn.microsoft.com/en-us/library/ms789031%28v=vs.110%29.aspx] > On the other hand for LDAP authentication, typically ‘Basic’ authentication > scheme is used (Note TLS is mandatory with Basic authentication scheme). > http://httpd.apache.org/docs/trunk/mod/mod_authnz_ldap.html > Hence for this feature, the idea would be to provide a custom implementation > of Hadoop AuthenticationHandler and Authenticator interfaces which would > support both schemes - Kerberos (via Negotiate auth challenge) and LDAP (via > Basic auth challenge). During the authentication phase, it would send both > the challenges and let client pick the appropriate one. If client responds > with an ‘Authorization’ header tagged with ‘Negotiate’ - it will use Kerberos > authentication. If client responds with an ‘Authorization’ header tagged with > ‘Basic’ - it will use LDAP authentication. > Note - some HTTP clients (e.g. curl or Apache Http Java client) need to be > configured to use one scheme over the other e.g. > - curl tool supports option to use either Kerberos (via --negotiate flag) or > username/password based authentication (via --basic and -u flags). > - Apache HttpClient library can be configured to use specific authentication > scheme. > http://hc.apache.org/httpcomponents-client-ga/tutorial/html/authentication.html > Typically web browsers automatically choose an authentication scheme based on > a notion of “strength” of security. e.g. take a look at the [design of Chrome > browser for HTTP > authentication|https://www.chromium.org/developers/design-documents/http-authentication] -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13706) Update jackson from 1.9.13 to 2.x in hadoop-common-project
[ https://issues.apache.org/jira/browse/HADOOP-13706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-13706: --- Status: Patch Available (was: Open) > Update jackson from 1.9.13 to 2.x in hadoop-common-project > -- > > Key: HADOOP-13706 > URL: https://issues.apache.org/jira/browse/HADOOP-13706 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: HADOOP-13706.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13706) Update jackson from 1.9.13 to 2.x in hadoop-common-project
[ https://issues.apache.org/jira/browse/HADOOP-13706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-13706: --- Attachment: HADOOP-13706.01.patch > Update jackson from 1.9.13 to 2.x in hadoop-common-project > -- > > Key: HADOOP-13706 > URL: https://issues.apache.org/jira/browse/HADOOP-13706 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: HADOOP-13706.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13708) Fix a few typos in site *.md documents
[ https://issues.apache.org/jira/browse/HADOOP-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570819#comment-15570819 ] ASF GitHub Bot commented on HADOOP-13708: - Github user danix800 closed the pull request at: https://github.com/apache/hadoop/pull/140 > Fix a few typos in site *.md documents > -- > > Key: HADOOP-13708 > URL: https://issues.apache.org/jira/browse/HADOOP-13708 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.8.0 >Reporter: Ding Fei >Assignee: Ding Fei >Priority: Minor > Attachments: HADOOP-13708-1.patch, HADOOP-13708-2.patch, > HADOOP-13708-3.patch, HADOOP-13708.patch > > > Fix several typos in site *.md documents. > Touched documents listed: > * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/notation.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md > * > hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md > * hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md > * hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13708) Fix a few typos in site *.md documents
[
https://issues.apache.org/jira/browse/HADOOP-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570815#comment-15570815
]
Hadoop QA commented on HADOOP-13708:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HADOOP-13708 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-13708 |
| GITHUB PR | https://github.com/apache/hadoop/pull/140 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10757/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix a few typos in site *.md documents
> --
>
> Key: HADOOP-13708
> URL: https://issues.apache.org/jira/browse/HADOOP-13708
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.8.0
>Reporter: Ding Fei
>Assignee: Ding Fei
>Priority: Minor
> Attachments: HADOOP-13708-1.patch, HADOOP-13708-2.patch,
> HADOOP-13708-3.patch, HADOOP-13708.patch
>
>
> Fix several typos in site *.md documents.
> Touched documents listed:
> * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md
> *
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
> *
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/notation.md
> *
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md
> *
> hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md
> * hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
> * hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13708) Fix a few typos in site *.md documents
[ https://issues.apache.org/jira/browse/HADOOP-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ding Fei updated HADOOP-13708: -- Attachment: HADOOP-13708-3.patch > Fix a few typos in site *.md documents > -- > > Key: HADOOP-13708 > URL: https://issues.apache.org/jira/browse/HADOOP-13708 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.8.0 >Reporter: Ding Fei >Assignee: Ding Fei >Priority: Minor > Attachments: HADOOP-13708-1.patch, HADOOP-13708-2.patch, > HADOOP-13708-3.patch, HADOOP-13708.patch > > > Fix several typos in site *.md documents. > Touched documents listed: > * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/notation.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md > * > hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md > * hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md > * hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13708) Fix a few typos in site *.md documents
[
https://issues.apache.org/jira/browse/HADOOP-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570677#comment-15570677
]
Hadoop QA commented on HADOOP-13708:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HADOOP-13708 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-13708 |
| GITHUB PR | https://github.com/apache/hadoop/pull/140 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10756/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix a few typos in site *.md documents
> --
>
> Key: HADOOP-13708
> URL: https://issues.apache.org/jira/browse/HADOOP-13708
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.8.0
>Reporter: Ding Fei
>Assignee: Ding Fei
>Priority: Minor
> Attachments: HADOOP-13708-1.patch, HADOOP-13708-2.patch,
> HADOOP-13708.patch
>
>
> Fix several typos in site *.md documents.
> Touched documents listed:
> * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md
> *
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
> *
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/notation.md
> *
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md
> *
> hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md
> * hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
> * hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13708) Fix a few typos in site *.md documents
[ https://issues.apache.org/jira/browse/HADOOP-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ding Fei updated HADOOP-13708: -- Attachment: HADOOP-13708-2.patch > Fix a few typos in site *.md documents > -- > > Key: HADOOP-13708 > URL: https://issues.apache.org/jira/browse/HADOOP-13708 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.8.0 >Reporter: Ding Fei >Assignee: Ding Fei >Priority: Minor > Attachments: HADOOP-13708-1.patch, HADOOP-13708-2.patch, > HADOOP-13708.patch > > > Fix several typos in site *.md documents. > Touched documents listed: > * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/notation.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md > * > hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md > * hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md > * hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13708) Fix a few typos in site *.md documents
[
https://issues.apache.org/jira/browse/HADOOP-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570656#comment-15570656
]
Hadoop QA commented on HADOOP-13708:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HADOOP-13708 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-13708 |
| GITHUB PR | https://github.com/apache/hadoop/pull/140 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10755/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix a few typos in site *.md documents
> --
>
> Key: HADOOP-13708
> URL: https://issues.apache.org/jira/browse/HADOOP-13708
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.8.0
>Reporter: Ding Fei
>Assignee: Ding Fei
>Priority: Minor
> Attachments: HADOOP-13708-1.patch, HADOOP-13708.patch
>
>
> Fix several typos in site *.md documents.
> Touched documents listed:
> * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md
> *
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
> *
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/notation.md
> *
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md
> *
> hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md
> * hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
> * hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13708) Fix a few typos in site *.md documents
[ https://issues.apache.org/jira/browse/HADOOP-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570654#comment-15570654 ] Ding Fei commented on HADOOP-13708: --- Totally confused by GITHUB pull request workflow :-(. Please use patch file only. > Fix a few typos in site *.md documents > -- > > Key: HADOOP-13708 > URL: https://issues.apache.org/jira/browse/HADOOP-13708 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.8.0 >Reporter: Ding Fei >Assignee: Ding Fei >Priority: Minor > Attachments: HADOOP-13708-1.patch, HADOOP-13708.patch > > > Fix several typos in site *.md documents. > Touched documents listed: > * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/notation.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md > * > hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md > * hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md > * hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13708) Fix a few typos in site *.md documents
[ https://issues.apache.org/jira/browse/HADOOP-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570650#comment-15570650 ] Ding Fei commented on HADOOP-13708: --- Thanks for suggestions! Updated patch uploaded! > Fix a few typos in site *.md documents > -- > > Key: HADOOP-13708 > URL: https://issues.apache.org/jira/browse/HADOOP-13708 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.8.0 >Reporter: Ding Fei >Assignee: Ding Fei >Priority: Minor > Attachments: HADOOP-13708-1.patch, HADOOP-13708.patch > > > Fix several typos in site *.md documents. > Touched documents listed: > * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/notation.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md > * > hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md > * hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md > * hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13708) Fix a few typos in site *.md documents
[ https://issues.apache.org/jira/browse/HADOOP-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ding Fei updated HADOOP-13708: -- Attachment: HADOOP-13708-1.patch > Fix a few typos in site *.md documents > -- > > Key: HADOOP-13708 > URL: https://issues.apache.org/jira/browse/HADOOP-13708 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.8.0 >Reporter: Ding Fei >Assignee: Ding Fei >Priority: Minor > Attachments: HADOOP-13708-1.patch, HADOOP-13708.patch > > > Fix several typos in site *.md documents. > Touched documents listed: > * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/notation.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md > * > hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md > * hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md > * hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13449) S3Guard: Implement DynamoDBMetadataStore.
[ https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HADOOP-13449: --- Attachment: HADOOP-13449-HADOOP-13345.wip.patch Attach a demo patch. I'm working on a real patch and will post it soon. Thanks. > S3Guard: Implement DynamoDBMetadataStore. > - > > Key: HADOOP-13449 > URL: https://issues.apache.org/jira/browse/HADOOP-13449 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Mingliang Liu > Attachments: HADOOP-13449-HADOOP-13345.wip.patch > > > Provide an implementation of the metadata store backed by DynamoDB. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12082) Support multiple authentication schemes via AuthenticationFilter
[ https://issues.apache.org/jira/browse/HADOOP-12082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570578#comment-15570578 ] Benoy Antony commented on HADOOP-12082: --- Looks good. Just a suggestion for documentation. I think , its more appropriate to change the filter-name in the example to "authFilter" instead "kerberosFilter" . > Support multiple authentication schemes via AuthenticationFilter > > > Key: HADOOP-12082 > URL: https://issues.apache.org/jira/browse/HADOOP-12082 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.6.0 >Reporter: Hrishikesh Gadre >Assignee: Hrishikesh Gadre > Attachments: HADOOP-12082-001.patch, HADOOP-12082.patch, > hadoop-ldap-auth-v2.patch, hadoop-ldap-auth-v3.patch, > hadoop-ldap-auth-v4.patch, hadoop-ldap-auth-v5.patch, > hadoop-ldap-auth-v6.patch, hadoop-ldap.patch, > multi-scheme-auth-support-poc.patch > > > The requirement is to support LDAP based authentication scheme via Hadoop > AuthenticationFilter. HADOOP-9054 added a support to plug-in custom > authentication scheme (in addition to Kerberos) via > AltKerberosAuthenticationHandler class. But it is based on selecting the > authentication mechanism based on User-Agent HTTP header which does not > conform to HTTP protocol semantics. > As per [RFC-2616|http://www.w3.org/Protocols/rfc2616/rfc2616.html] > - HTTP protocol provides a simple challenge-response authentication mechanism > that can be used by a server to challenge a client request and by a client to > provide the necessary authentication information. > - This mechanism is initiated by server sending the 401 (Authenticate) > response with ‘WWW-Authenticate’ header which includes at least one challenge > that indicates the authentication scheme(s) and parameters applicable to the > Request-URI. > - In case server supports multiple authentication schemes, it may return > multiple challenges with a 401 (Authenticate) response, and each challenge > may use a different auth-scheme. > - A user agent MUST choose to use the strongest auth-scheme it understands > and request credentials from the user based upon that challenge. > The existing Hadoop authentication filter implementation supports Kerberos > authentication scheme and uses ‘Negotiate’ as the challenge as part of > ‘WWW-Authenticate’ response header. As per the following documentation, > ‘Negotiate’ challenge scheme is only applicable to Kerberos (and Windows > NTLM) authentication schemes. > [SPNEGO-based Kerberos and NTLM HTTP > Authentication|http://tools.ietf.org/html/rfc4559] > [Understanding HTTP > Authentication|https://msdn.microsoft.com/en-us/library/ms789031%28v=vs.110%29.aspx] > On the other hand for LDAP authentication, typically ‘Basic’ authentication > scheme is used (Note TLS is mandatory with Basic authentication scheme). > http://httpd.apache.org/docs/trunk/mod/mod_authnz_ldap.html > Hence for this feature, the idea would be to provide a custom implementation > of Hadoop AuthenticationHandler and Authenticator interfaces which would > support both schemes - Kerberos (via Negotiate auth challenge) and LDAP (via > Basic auth challenge). During the authentication phase, it would send both > the challenges and let client pick the appropriate one. If client responds > with an ‘Authorization’ header tagged with ‘Negotiate’ - it will use Kerberos > authentication. If client responds with an ‘Authorization’ header tagged with > ‘Basic’ - it will use LDAP authentication. > Note - some HTTP clients (e.g. curl or Apache Http Java client) need to be > configured to use one scheme over the other e.g. > - curl tool supports option to use either Kerberos (via --negotiate flag) or > username/password based authentication (via --basic and -u flags). > - Apache HttpClient library can be configured to use specific authentication > scheme. > http://hc.apache.org/httpcomponents-client-ga/tutorial/html/authentication.html > Typically web browsers automatically choose an authentication scheme based on > a notion of “strength” of security. e.g. take a look at the [design of Chrome > browser for HTTP > authentication|https://www.chromium.org/developers/design-documents/http-authentication] -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13702) Add a new instrumented read-write lock
[ https://issues.apache.org/jira/browse/HADOOP-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570530#comment-15570530 ] Jingcheng Du commented on HADOOP-13702: --- Hi guys, any comments for the latest patch? All of them are appreciated. > Add a new instrumented read-write lock > -- > > Key: HADOOP-13702 > URL: https://issues.apache.org/jira/browse/HADOOP-13702 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: HADOOP-13702-V6.patch, HDFS-10924-2.patch, > HDFS-10924-3.patch, HDFS-10924-4.patch, HDFS-10924-5.patch, HDFS-10924.patch > > > Add a new instrumented read-write lock in hadoop common, so that the > HDFS-9668 can use this to improve the locking in FsDatasetImpl -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13707) If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot be accessed
[
https://issues.apache.org/jira/browse/HADOOP-13707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570527#comment-15570527
]
Hadoop QA commented on HADOOP-13707:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
47s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13707 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12833010/HADOOP-13707.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a8353e013568 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 12d739a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10754/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10754/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot
> be accessed
> -
>
> Key: HADOOP-13707
> URL: https://issues.apache.org/jira/browse/HADOOP-13707
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yuanbo Liu
>Assignee: Yuanbo Liu
> Labels: security
> Attachments: HADOOP-13707.001.patch, HADOOP-13707.002.patch,
> HADOOP-13707.003.patch, HADOOP-13707.004.patch
>
>
> In
[jira] [Updated] (HADOOP-13714) Tighten up our compatibility guidelines for Hadoop 3
[ https://issues.apache.org/jira/browse/HADOOP-13714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-13714: -- Target Version/s: 3.0.0-beta1 (was: 3.0.0-alpha2) > Tighten up our compatibility guidelines for Hadoop 3 > > > Key: HADOOP-13714 > URL: https://issues.apache.org/jira/browse/HADOOP-13714 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.3 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Blocker > > Our current compatibility guidelines are incomplete and loose. For many > categories, we do not have a policy. It would be nice to actually define > those policies so our users know what to expect and the developers know what > releases to target their changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13718) There is no filterInitializer for initializing DelegationTokenAuthenticationFilter in Hadoop common
[ https://issues.apache.org/jira/browse/HADOOP-13718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanbo Liu resolved HADOOP-13718. - Resolution: Duplicate > There is no filterInitializer for initializing > DelegationTokenAuthenticationFilter in Hadoop common > --- > > Key: HADOOP-13718 > URL: https://issues.apache.org/jira/browse/HADOOP-13718 > Project: Hadoop Common > Issue Type: Improvement > Environment: There is no filterInitializer for initializing > DelegationTokenFilter in Hadoop common. Yarn implement its own > filterInitializer RMAuthenticationFilterInitializer to add > DelegationTokenFilter to support proxy user and delegation token > authentication. >Reporter: Shi Wang > > There is no filterInitializer for initializing DelegationTokenFilter in > Hadoop common. > Yarn implement its own filterInitializer RMAuthenticationFilterInitializer to > add DelegationTokenFilter to support proxy user and delegation token > authentication. > This is useful to use DelegationTokenAuthenticationFilter for HADOOP Web > Console authentication. Especially after KNOX-565, all quick link could go > through knox and we'll need DelegationTokenAuthenticationFilter to do the > "getDoAsUser" job and call kerberosAuthenticationHandler for us. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13718) There is no filterInitializer for initializing DelegationTokenAuthenticationFilter in Hadoop common
[ https://issues.apache.org/jira/browse/HADOOP-13718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570441#comment-15570441 ] Yuanbo Liu commented on HADOOP-13718: - [~Wancy] Thanks to file this issue. I'd like to fix it in HADOOP-13119. Eric didn't want to lose discussion there for filing another jira. I'm gonna mark this jira as duplicate. If you have any suggestion, please let me know. Thanks a lot! > There is no filterInitializer for initializing > DelegationTokenAuthenticationFilter in Hadoop common > --- > > Key: HADOOP-13718 > URL: https://issues.apache.org/jira/browse/HADOOP-13718 > Project: Hadoop Common > Issue Type: Improvement > Environment: There is no filterInitializer for initializing > DelegationTokenFilter in Hadoop common. Yarn implement its own > filterInitializer RMAuthenticationFilterInitializer to add > DelegationTokenFilter to support proxy user and delegation token > authentication. >Reporter: Shi Wang > > There is no filterInitializer for initializing DelegationTokenFilter in > Hadoop common. > Yarn implement its own filterInitializer RMAuthenticationFilterInitializer to > add DelegationTokenFilter to support proxy user and delegation token > authentication. > This is useful to use DelegationTokenAuthenticationFilter for HADOOP Web > Console authentication. Especially after KNOX-565, all quick link could go > through knox and we'll need DelegationTokenAuthenticationFilter to do the > "getDoAsUser" job and call kerberosAuthenticationHandler for us. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11656) Classpath isolation for downstream clients
[ https://issues.apache.org/jira/browse/HADOOP-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-11656: - Target Version/s: 3.0.0-alpha2 (was: ) > Classpath isolation for downstream clients > -- > > Key: HADOOP-11656 > URL: https://issues.apache.org/jira/browse/HADOOP-11656 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Sean Busbey >Assignee: Sean Busbey > Labels: classloading, classpath, dependencies, scripts, shell > Attachments: HADOOP-11656_proposal.md > > > Currently, Hadoop exposes downstream clients to a variety of third party > libraries. As our code base grows and matures we increase the set of > libraries we rely on. At the same time, as our user base grows we increase > the likelihood that some downstream project will run into a conflict while > attempting to use a different version of some library we depend on. This has > already happened with i.e. Guava several times for HBase, Accumulo, and Spark > (and I'm sure others). > While YARN-286 and MAPREDUCE-1700 provided an initial effort, they default to > off and they don't do anything to help dependency conflicts on the driver > side or for folks talking to HDFS directly. This should serve as an umbrella > for changes needed to do things thoroughly on the next major version. > We should ensure that downstream clients > 1) can depend on a client artifact for each of HDFS, YARN, and MapReduce that > doesn't pull in any third party dependencies > 2) only see our public API classes (or as close to this as feasible) when > executing user provided code, whether client side in a launcher/driver or on > the cluster in a container or within MR. > This provides us with a double benefit: users get less grief when they want > to run substantially ahead or behind the versions we need and the project is > freer to change our own dependency versions because they'll no longer be in > our compatibility promises. > Project specific task jiras to follow after I get some justifying use cases > written in the comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13714) Tighten up our compatibility guidelines for Hadoop 3
[ https://issues.apache.org/jira/browse/HADOOP-13714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570435#comment-15570435 ] Andrew Wang commented on HADOOP-13714: -- One major item in if we finish HADOOP-11656 is some guarantees around not breaking the client classpath. > Tighten up our compatibility guidelines for Hadoop 3 > > > Key: HADOOP-13714 > URL: https://issues.apache.org/jira/browse/HADOOP-13714 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.3 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Blocker > > Our current compatibility guidelines are incomplete and loose. For many > categories, we do not have a policy. It would be nice to actually define > those policies so our users know what to expect and the developers know what > releases to target their changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13707) If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot be accessed
[
https://issues.apache.org/jira/browse/HADOOP-13707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuanbo Liu updated HADOOP-13707:
Attachment: HADOOP-13707.004.patch
uploaded v4 patch to address checkstyle issue
> If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot
> be accessed
> -
>
> Key: HADOOP-13707
> URL: https://issues.apache.org/jira/browse/HADOOP-13707
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yuanbo Liu
>Assignee: Yuanbo Liu
> Labels: security
> Attachments: HADOOP-13707.001.patch, HADOOP-13707.002.patch,
> HADOOP-13707.003.patch, HADOOP-13707.004.patch
>
>
> In {{HttpServer2#hasAdministratorAccess}}, it uses
> `hadoop.security.authorization` to detect whether HTTP is authenticated.
> It's not correct, because enabling Kerberos and HTTP SPNEGO are two steps. If
> Kerberos is enabled while HTTP SPNEGO is not, some links cannot be accessed,
> such as "/logs", and it will return error message as below:
> {quote}
> HTTP ERROR 403
> Problem accessing /logs/. Reason:
> User dr.who is unauthorized to access this page.
> {quote}
> We should make sure {{HttpServletRequest#getAuthType}} is not null before we
> invoke {{HttpServer2#hasAdministratorAccess}}.
> {{getAuthType}} means to get the authorization scheme of this request
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13024) Distcp with -delete feature on raw data not implemented
[
https://issues.apache.org/jira/browse/HADOOP-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570368#comment-15570368
]
Hadoop QA commented on HADOOP-13024:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
3s{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 11s{color} | {color:orange} hadoop-tools/hadoop-distcp: The patch generated
1 new + 71 unchanged - 2 fixed = 72 total (was 73) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
45s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13024 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12833006/HADOOP-13024.patch.8 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 765d80310532 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 12d739a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10753/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10753/testReport/ |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10753/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Distcp with -delete feature on raw data not implemented
> ---
>
> Key:
[jira] [Commented] (HADOOP-13024) Distcp with -delete feature on raw data not implemented
[
https://issues.apache.org/jira/browse/HADOOP-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570323#comment-15570323
]
Hadoop QA commented on HADOOP-13024:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
1s{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 11s{color} | {color:orange} hadoop-tools/hadoop-distcp: The patch generated
3 new + 71 unchanged - 2 fixed = 74 total (was 73) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
3s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13024 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12833000/HADOOP-13024.patch.7 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux da1aab232365 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 12d739a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10752/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10752/testReport/ |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10752/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Distcp with -delete feature on raw data not implemented
> ---
>
> Key:
[jira] [Commented] (HADOOP-11798) Native raw erasure coder in XOR codes
[ https://issues.apache.org/jira/browse/HADOOP-11798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570316#comment-15570316 ] Kai Zheng commented on HADOOP-11798: Another idea is, better to add inter-operable tests for this native xor coder against the existing Java version. > Native raw erasure coder in XOR codes > - > > Key: HADOOP-11798 > URL: https://issues.apache.org/jira/browse/HADOOP-11798 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: SammiChen > Labels: hdfs-ec-3.0-must-do > Fix For: HDFS-7285 > > Attachments: HADOOP-11798-v1.patch, HADOOP-11798-v2.patch > > > Raw XOR coder is utilized in Reed-Solomon erasure coder in an optimization to > recover only one erased block which is in most often case. It can also be > used in HitchHiker coder. Therefore a native implementation of it would be > deserved for performance gain. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13024) Distcp with -delete feature on raw data not implemented
[
https://issues.apache.org/jira/browse/HADOOP-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mavin Martin updated HADOOP-13024:
--
Attachment: HADOOP-13024.patch.8
Fixed checkstyle
> Distcp with -delete feature on raw data not implemented
> ---
>
> Key: HADOOP-13024
> URL: https://issues.apache.org/jira/browse/HADOOP-13024
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Mavin Martin
>Assignee: Mavin Martin
> Attachments: HADOOP-13024.patch, HADOOP-13024.patch,
> HADOOP-13024.patch.3, HADOOP-13024.patch.4, HADOOP-13024.patch.5,
> HADOOP-13024.patch.6, HADOOP-13024.patch.7, HADOOP-13024.patch.8
>
>
> When doing distcp of raw data using -delete feature, following bug appears.
> {code}
> [root@xxx bin]# hadoop distcp -delete -update /.reserved/raw/tmp/a
> /.reserved/raw/tmp/b
> 16/04/14 02:54:01 ERROR tools.DistCp: Exception encountered
> java.io.IOException: DistCp failure: Job job_xxx has failed: Job commit
> failed: org.apache.hadoop.tools.CopyListing$InvalidInputException: The source
> path 'hdfs://nn/.reserved/raw/tmp/b' starts with /.reserved/raw but the
> target path 'hdfs://nn/NONE' does not. Either all or none of the paths must
> have this prefix.
> at
> org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:141)
> at
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:85)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
> at
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.deleteMissing(CopyCommitter.java:244)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:94)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:187)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:122)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:429)
> {code}
> The issue is not with the distributed copy, the issue is when it tries to
> delete things in the target that no longer exist in the source, it
> revalidates to make sure NONE is in the /.reserved/raw domain.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13024) Distcp with -delete feature on raw data not implemented
[
https://issues.apache.org/jira/browse/HADOOP-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570272#comment-15570272
]
Hadoop QA commented on HADOOP-13024:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
1s{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 11s{color} | {color:orange} hadoop-tools/hadoop-distcp: The patch generated
3 new + 71 unchanged - 2 fixed = 74 total (was 73) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
55s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13024 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832999/HADOOP-13024.patch.6 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 754aa9e32f36 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 12d739a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10751/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10751/testReport/ |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10751/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Distcp with -delete feature on raw data not implemented
> ---
>
> Key:
[jira] [Commented] (HADOOP-13669) KMS Server should log exceptions before throwing
[
https://issues.apache.org/jira/browse/HADOOP-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570245#comment-15570245
]
Hadoop QA commented on HADOOP-13669:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
26s{color} | {color:red} hadoop-common-project/hadoop-kms in trunk has 2 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
9s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13669 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832943/trigger.patch |
| Optional Tests | asflicense xml compile javac javadoc mvninstall
mvnsite unit findbugs checkstyle |
| uname | Linux 27fb6be82c73 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 12d739a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10750/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-kms-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10750/testReport/ |
| modules | C: hadoop-common-project/hadoop-kms U:
hadoop-common-project/hadoop-kms |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10750/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> KMS Server should log exceptions before throwing
>
>
> Key:
[jira] [Updated] (HADOOP-13024) Distcp with -delete feature on raw data not implemented
[
https://issues.apache.org/jira/browse/HADOOP-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mavin Martin updated HADOOP-13024:
--
Attachment: HADOOP-13024.patch.7
Extracted common workflow tests to a common method.
> Distcp with -delete feature on raw data not implemented
> ---
>
> Key: HADOOP-13024
> URL: https://issues.apache.org/jira/browse/HADOOP-13024
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Mavin Martin
>Assignee: Mavin Martin
> Attachments: HADOOP-13024.patch, HADOOP-13024.patch,
> HADOOP-13024.patch.3, HADOOP-13024.patch.4, HADOOP-13024.patch.5,
> HADOOP-13024.patch.6, HADOOP-13024.patch.7
>
>
> When doing distcp of raw data using -delete feature, following bug appears.
> {code}
> [root@xxx bin]# hadoop distcp -delete -update /.reserved/raw/tmp/a
> /.reserved/raw/tmp/b
> 16/04/14 02:54:01 ERROR tools.DistCp: Exception encountered
> java.io.IOException: DistCp failure: Job job_xxx has failed: Job commit
> failed: org.apache.hadoop.tools.CopyListing$InvalidInputException: The source
> path 'hdfs://nn/.reserved/raw/tmp/b' starts with /.reserved/raw but the
> target path 'hdfs://nn/NONE' does not. Either all or none of the paths must
> have this prefix.
> at
> org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:141)
> at
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:85)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
> at
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.deleteMissing(CopyCommitter.java:244)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:94)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:187)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:122)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:429)
> {code}
> The issue is not with the distributed copy, the issue is when it tries to
> delete things in the target that no longer exist in the source, it
> revalidates to make sure NONE is in the /.reserved/raw domain.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13024) Distcp with -delete feature on raw data not implemented
[
https://issues.apache.org/jira/browse/HADOOP-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mavin Martin updated HADOOP-13024:
--
Attachment: HADOOP-13024.patch.6
Thanks for the feedback Jing. I have made the changes and added a unit test.
> Distcp with -delete feature on raw data not implemented
> ---
>
> Key: HADOOP-13024
> URL: https://issues.apache.org/jira/browse/HADOOP-13024
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Mavin Martin
>Assignee: Mavin Martin
> Attachments: HADOOP-13024.patch, HADOOP-13024.patch,
> HADOOP-13024.patch.3, HADOOP-13024.patch.4, HADOOP-13024.patch.5,
> HADOOP-13024.patch.6
>
>
> When doing distcp of raw data using -delete feature, following bug appears.
> {code}
> [root@xxx bin]# hadoop distcp -delete -update /.reserved/raw/tmp/a
> /.reserved/raw/tmp/b
> 16/04/14 02:54:01 ERROR tools.DistCp: Exception encountered
> java.io.IOException: DistCp failure: Job job_xxx has failed: Job commit
> failed: org.apache.hadoop.tools.CopyListing$InvalidInputException: The source
> path 'hdfs://nn/.reserved/raw/tmp/b' starts with /.reserved/raw but the
> target path 'hdfs://nn/NONE' does not. Either all or none of the paths must
> have this prefix.
> at
> org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:141)
> at
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:85)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
> at
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.deleteMissing(CopyCommitter.java:244)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:94)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:187)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:122)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:429)
> {code}
> The issue is not with the distributed copy, the issue is when it tries to
> delete things in the target that no longer exist in the source, it
> revalidates to make sure NONE is in the /.reserved/raw domain.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13718) There is no filterInitializer for initializing DelegationTokenAuthenticationFilter in Hadoop common
[ https://issues.apache.org/jira/browse/HADOOP-13718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shi Wang updated HADOOP-13718: -- Description: There is no filterInitializer for initializing DelegationTokenFilter in Hadoop common. Yarn implement its own filterInitializer RMAuthenticationFilterInitializer to add DelegationTokenFilter to support proxy user and delegation token authentication. This is useful to use DelegationTokenAuthenticationFilter for HADOOP Web Console authentication. Especially after KNOX-565, all quick link could go through knox and we'll need DelegationTokenAuthenticationFilter to do the "getDoAsUser" job and call kerberosAuthenticationHandler for us. was: There is no filterInitializer for initializing DelegationTokenFilter in Hadoop common. Yarn implement its own filterInitializer RMAuthenticationFilterInitializer to add DelegationTokenFilter to support proxy user and delegation token authentication. This is useful to use DelegationTokenAuthenticationFilter for HADOOP Web Console authentication. Especially after KNOX-565, all quick link could go through knox and we'll need > There is no filterInitializer for initializing > DelegationTokenAuthenticationFilter in Hadoop common > --- > > Key: HADOOP-13718 > URL: https://issues.apache.org/jira/browse/HADOOP-13718 > Project: Hadoop Common > Issue Type: Improvement > Environment: There is no filterInitializer for initializing > DelegationTokenFilter in Hadoop common. Yarn implement its own > filterInitializer RMAuthenticationFilterInitializer to add > DelegationTokenFilter to support proxy user and delegation token > authentication. >Reporter: Shi Wang > > There is no filterInitializer for initializing DelegationTokenFilter in > Hadoop common. > Yarn implement its own filterInitializer RMAuthenticationFilterInitializer to > add DelegationTokenFilter to support proxy user and delegation token > authentication. > This is useful to use DelegationTokenAuthenticationFilter for HADOOP Web > Console authentication. Especially after KNOX-565, all quick link could go > through knox and we'll need DelegationTokenAuthenticationFilter to do the > "getDoAsUser" job and call kerberosAuthenticationHandler for us. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13710) Supress CachingGetSpaceUsed from logging interrupted exception stacktrace
[
https://issues.apache.org/jira/browse/HADOOP-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570158#comment-15570158
]
Wei-Chiu Chuang commented on HADOOP-13710:
--
+1 pending jenkins.
> Supress CachingGetSpaceUsed from logging interrupted exception stacktrace
> -
>
> Key: HADOOP-13710
> URL: https://issues.apache.org/jira/browse/HADOOP-13710
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Hanisha Koneru
>Priority: Minor
> Labels: supportability
> Attachments: HDFS-13710.000.patch
>
>
> CachingGetSpaceUsed thread is typically interrupted when the node is
> shutdown. Since this is a routine operation, there seems not much value to
> print the stacktrace of an {{InterruptedException}}.
> {quote}
> 2016-10-11 10:02:25,894 WARN fs.CachingGetSpaceUsed
> (CachingGetSpaceUsed.java:run(180)) - Thread Interrupted waiting to refresh
> disk information
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:176)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13718) There is no filterInitializer for initializing DelegationTokenAuthenticationFilter in Hadoop common
[ https://issues.apache.org/jira/browse/HADOOP-13718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shi Wang updated HADOOP-13718: -- Description: There is no filterInitializer for initializing DelegationTokenFilter in Hadoop common. Yarn implement its own filterInitializer RMAuthenticationFilterInitializer to add DelegationTokenFilter to support proxy user and delegation token authentication. This is useful to use DelegationTokenAuthenticationFilter for HADOOP Web Console authentication. Especially after KNOX-565, all quick link could go through knox and we'll need was: There is no filterInitializer for initializing DelegationTokenFilter in Hadoop common. Yarn implement its own filterInitializer RMAuthenticationFilterInitializer to add DelegationTokenFilter to support proxy user and delegation token authentication. This is > There is no filterInitializer for initializing > DelegationTokenAuthenticationFilter in Hadoop common > --- > > Key: HADOOP-13718 > URL: https://issues.apache.org/jira/browse/HADOOP-13718 > Project: Hadoop Common > Issue Type: Improvement > Environment: There is no filterInitializer for initializing > DelegationTokenFilter in Hadoop common. Yarn implement its own > filterInitializer RMAuthenticationFilterInitializer to add > DelegationTokenFilter to support proxy user and delegation token > authentication. >Reporter: Shi Wang > > There is no filterInitializer for initializing DelegationTokenFilter in > Hadoop common. > Yarn implement its own filterInitializer RMAuthenticationFilterInitializer to > add DelegationTokenFilter to support proxy user and delegation token > authentication. > This is useful to use DelegationTokenAuthenticationFilter for HADOOP Web > Console authentication. Especially after KNOX-565, all quick link could go > through knox and we'll need -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12774) s3a should use UGI.getCurrentUser.getShortname() for username
[
https://issues.apache.org/jira/browse/HADOOP-12774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570148#comment-15570148
]
Hadoop QA commented on HADOOP-12774:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
36s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 2
new + 9 unchanged - 0 fixed = 11 total (was 9) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
20s{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_111.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HADOOP-12774 |
| GITHUB PR | https://github.com/apache/hadoop/pull/136 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 19b394eb8f95 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / f131d61 |
| Default Java | 1.7.0_111 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_101
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 |
| findbugs | v3.0.0 |
|
[jira] [Updated] (HADOOP-13718) There is no filterInitializer for initializing DelegationTokenAuthenticationFilter in Hadoop common
[ https://issues.apache.org/jira/browse/HADOOP-13718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shi Wang updated HADOOP-13718: -- Description: There is no filterInitializer for initializing DelegationTokenFilter in Hadoop common. Yarn implement its own filterInitializer RMAuthenticationFilterInitializer to add DelegationTokenFilter to support proxy user and delegation token authentication. This is > There is no filterInitializer for initializing > DelegationTokenAuthenticationFilter in Hadoop common > --- > > Key: HADOOP-13718 > URL: https://issues.apache.org/jira/browse/HADOOP-13718 > Project: Hadoop Common > Issue Type: Improvement > Environment: There is no filterInitializer for initializing > DelegationTokenFilter in Hadoop common. Yarn implement its own > filterInitializer RMAuthenticationFilterInitializer to add > DelegationTokenFilter to support proxy user and delegation token > authentication. >Reporter: Shi Wang > > There is no filterInitializer for initializing DelegationTokenFilter in > Hadoop common. > Yarn implement its own filterInitializer RMAuthenticationFilterInitializer to > add DelegationTokenFilter to support proxy user and delegation token > authentication. > This is -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13718) There is no filterInitializer for initializing DelegationTokenAuthenticationFilter in Hadoop common
[ https://issues.apache.org/jira/browse/HADOOP-13718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shi Wang updated HADOOP-13718: -- Summary: There is no filterInitializer for initializing DelegationTokenAuthenticationFilter in Hadoop common (was: There is no filterInitializer for initializing DelegationTokenFilter in Hadoop common) > There is no filterInitializer for initializing > DelegationTokenAuthenticationFilter in Hadoop common > --- > > Key: HADOOP-13718 > URL: https://issues.apache.org/jira/browse/HADOOP-13718 > Project: Hadoop Common > Issue Type: Improvement > Environment: There is no filterInitializer for initializing > DelegationTokenFilter in Hadoop common. Yarn implement its own > filterInitializer RMAuthenticationFilterInitializer to add > DelegationTokenFilter to support proxy user and delegation token > authentication. >Reporter: Shi Wang > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13718) There is no filterInitializer for initializing DelegationTokenFilter in Hadoop common
Shi Wang created HADOOP-13718: - Summary: There is no filterInitializer for initializing DelegationTokenFilter in Hadoop common Key: HADOOP-13718 URL: https://issues.apache.org/jira/browse/HADOOP-13718 Project: Hadoop Common Issue Type: Improvement Environment: There is no filterInitializer for initializing DelegationTokenFilter in Hadoop common. Yarn implement its own filterInitializer RMAuthenticationFilterInitializer to add DelegationTokenFilter to support proxy user and delegation token authentication. Reporter: Shi Wang -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12774) s3a should use UGI.getCurrentUser.getShortname() for username
[
https://issues.apache.org/jira/browse/HADOOP-12774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570116#comment-15570116
]
Chris Nauroth commented on HADOOP-12774:
The latest pull request looks good to me. I think we still have an unused
import of {{PrivilegedAction}} in {{ITestS3AConfiguration}}. That's easy to
clean up on commit. I submitted another Jenkins run here:
https://builds.apache.org/view/PreCommit%20Builds/job/PreCommit-HADOOP-Build/10749/
> s3a should use UGI.getCurrentUser.getShortname() for username
> -
>
> Key: HADOOP-12774
> URL: https://issues.apache.org/jira/browse/HADOOP-12774
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> S3a users {{System.getProperty("user.name")}} to get the username for the
> homedir. This is wrong, as it doesn't work on a YARN app where the identity
> is set by HADOOP_USER_NAME, or in a doAs clause.
> Obviously, {{UGI.getCurrentUser.getShortname()}} provides that name,
> everywhere.
> This is a simple change in the source, though testing is harder ... probably
> best to try in a doAs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13700) Remove unthrown IOException from TrashPolicy#initialize and #getInstance signatures
[ https://issues.apache.org/jira/browse/HADOOP-13700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570107#comment-15570107 ] Hudson commented on HADOOP-13700: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10598 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10598/]) HADOOP-13700. Remove unthrown IOException from TrashPolicy#initialize (wang: rev 12d739a34ba868b3f7f5adf7f37a60d4aca9061b) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicy.java > Remove unthrown IOException from TrashPolicy#initialize and #getInstance > signatures > --- > > Key: HADOOP-13700 > URL: https://issues.apache.org/jira/browse/HADOOP-13700 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Andrew Wang >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13700.001.patch > > > TrashPolicy is marked as public & evolving, but its public API, specifically > TrashPolicy.getInstance() has been changed in an incompatible way. > 1) The path parameter is removed in 3.0 > 2) A new IOException is thrown in 3.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13717) Shell scripts call hadoop_verify_logdir even when command is not started as daemon
[
https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570101#comment-15570101
]
Andrew Wang commented on HADOOP-13717:
--
For a bit more context, we have some code that starts the balancer in the
foreground, without a log4j.properties file or setting $HADOOP_LOG_DIR.
verify_logdir then checks the default location ($HADOOP_HOME/logs), which is
not writable, and fails.
Other commands that do not support daemonization skip all these checks. In this
case, I'm not trying to run the balancer as a daemon. Is it reasonable to also
skip these checks in this situation? Looking more at the code, I think what I
want is to go directly to hadoop_java_exec rather than the daemon logic.
> Shell scripts call hadoop_verify_logdir even when command is not started as
> daemon
> --
>
> Key: HADOOP-13717
> URL: https://issues.apache.org/jira/browse/HADOOP-13717
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the
> "default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which
> specifies the log location isn't even used here, since the command is being
> started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into
> {{hadoop_start_daemon_wrapper}} instead, which does use the outfile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13710) Supress CachingGetSpaceUsed from logging interrupted exception stacktrace
[
https://issues.apache.org/jira/browse/HADOOP-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanisha Koneru updated HADOOP-13710:
Attachment: HDFS-13710.000.patch
> Supress CachingGetSpaceUsed from logging interrupted exception stacktrace
> -
>
> Key: HADOOP-13710
> URL: https://issues.apache.org/jira/browse/HADOOP-13710
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Hanisha Koneru
>Priority: Minor
> Labels: supportability
> Attachments: HDFS-13710.000.patch
>
>
> CachingGetSpaceUsed thread is typically interrupted when the node is
> shutdown. Since this is a routine operation, there seems not much value to
> print the stacktrace of an {{InterruptedException}}.
> {quote}
> 2016-10-11 10:02:25,894 WARN fs.CachingGetSpaceUsed
> (CachingGetSpaceUsed.java:run(180)) - Thread Interrupted waiting to refresh
> disk information
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:176)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13717) Shell scripts call hadoop_verify_logdir even when command is not started as daemon
[
https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570046#comment-15570046
]
Allen Wittenauer edited comment on HADOOP-13717 at 10/12/16 10:23 PM:
--
The log directory may still get used, depending upon the log4j settings.
(An example would be the HDFS audit log.)
was (Author: aw):
The log directory may still get used, depending upon the log4j settings.
> Shell scripts call hadoop_verify_logdir even when command is not started as
> daemon
> --
>
> Key: HADOOP-13717
> URL: https://issues.apache.org/jira/browse/HADOOP-13717
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the
> "default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which
> specifies the log location isn't even used here, since the command is being
> started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into
> {{hadoop_start_daemon_wrapper}} instead, which does use the outfile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13700) Remove unthrown IOException from TrashPolicy#initialize and #getInstance signatures
[ https://issues.apache.org/jira/browse/HADOOP-13700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-13700: - Resolution: Fixed Fix Version/s: 3.0.0-alpha2 2.8.0 Status: Resolved (was: Patch Available) Committed to trunk, branch-2, branch-2.8. Thanks Haibo for reporting and reviewing, Eddy for the final +1, and other watchers for discussion. > Remove unthrown IOException from TrashPolicy#initialize and #getInstance > signatures > --- > > Key: HADOOP-13700 > URL: https://issues.apache.org/jira/browse/HADOOP-13700 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Andrew Wang >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13700.001.patch > > > TrashPolicy is marked as public & evolving, but its public API, specifically > TrashPolicy.getInstance() has been changed in an incompatible way. > 1) The path parameter is removed in 3.0 > 2) A new IOException is thrown in 3.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13717) Shell scripts call hadoop_verify_logdir even when command is not started as daemon
[
https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570046#comment-15570046
]
Allen Wittenauer commented on HADOOP-13717:
---
The log directory may still get used, depending upon the log4j settings.
> Shell scripts call hadoop_verify_logdir even when command is not started as
> daemon
> --
>
> Key: HADOOP-13717
> URL: https://issues.apache.org/jira/browse/HADOOP-13717
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the
> "default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which
> specifies the log location isn't even used here, since the command is being
> started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into
> {{hadoop_start_daemon_wrapper}} instead, which does use the outfile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13717) Shell scripts call hadoop_verify_logdir even when command is not started as daemon
[
https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570041#comment-15570041
]
Andrew Wang commented on HADOOP-13717:
--
[~aw], does the problem in the JIRA description make sense to you? If so, I can
try a simple patch.
> Shell scripts call hadoop_verify_logdir even when command is not started as
> daemon
> --
>
> Key: HADOOP-13717
> URL: https://issues.apache.org/jira/browse/HADOOP-13717
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the
> "default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which
> specifies the log location isn't even used here, since the command is being
> started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into
> {{hadoop_start_daemon_wrapper}} instead, which does use the outfile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13717) Shell scripts call hadoop_verify_logdir even when command is not started as daemon
Andrew Wang created HADOOP-13717:
Summary: Shell scripts call hadoop_verify_logdir even when command
is not started as daemon
Key: HADOOP-13717
URL: https://issues.apache.org/jira/browse/HADOOP-13717
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 3.0.0-alpha1
Reporter: Andrew Wang
Issue found when working with the HDFS balancer.
In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the
"default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which
specifies the log location isn't even used here, since the command is being
started in the foreground.
I think we can push the {{hadoop_verify_logdir}} call down into
{{hadoop_start_daemon_wrapper}} instead, which does use the outfile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13234) Get random port by new ServerSocket(0).getLocalPort() in ServerSocketUtil#getPort
[
https://issues.apache.org/jira/browse/HADOOP-13234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569990#comment-15569990
]
Masatake Iwasaki commented on HADOOP-13234:
---
Since {{TestDFSShell#testMoveWithTargetPortEmpty}} has to use the default port
number, {{ServerSocketUtil#getPort}} can't be help.
> Get random port by new ServerSocket(0).getLocalPort() in
> ServerSocketUtil#getPort
> -
>
> Key: HADOOP-13234
> URL: https://issues.apache.org/jira/browse/HADOOP-13234
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HADOOP-13234-002.patch, HADOOP-13234.patch
>
>
> As per [~iwasakims] comment from
> [here|https://issues.apache.org/jira/browse/HDFS-10367?focusedCommentId=15275604=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15275604]
> we can get available random port by {{new ServerSocket(0).getLocalPort()}}
> and it's more portable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13234) Get random port by new ServerSocket(0).getLocalPort() in ServerSocketUtil#getPort
[
https://issues.apache.org/jira/browse/HADOOP-13234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569974#comment-15569974
]
Masatake Iwasaki commented on HADOOP-13234:
---
Thanks for the patch, [~brahmareddy].
I think manual free port searching has an advantage I have not noticed in
HDFS-10367.
{{ServerSocket(0).getLocalPort()}} always returns the number in ephemeral port
range based on the platform (e.g. 32768-61000 on Linux and 49152-65535 on
Windows). The current {{getPort{p, retries)}} tries the range between p and
65535 which is wider than epehmeral port range. It could be useful to reduce
possibility of port collision since client socket always uses ephemeral port.
I don't have motivation to replace existing code now baced on the fact above.
Can you propose additional advantage of using
{{ServerSocket(0).getLocalPort()}}?
> Get random port by new ServerSocket(0).getLocalPort() in
> ServerSocketUtil#getPort
> -
>
> Key: HADOOP-13234
> URL: https://issues.apache.org/jira/browse/HADOOP-13234
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HADOOP-13234-002.patch, HADOOP-13234.patch
>
>
> As per [~iwasakims] comment from
> [here|https://issues.apache.org/jira/browse/HDFS-10367?focusedCommentId=15275604=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15275604]
> we can get available random port by {{new ServerSocket(0).getLocalPort()}}
> and it's more portable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13710) Supress CachingGetSpaceUsed from logging interrupted exception stacktrace
[
https://issues.apache.org/jira/browse/HADOOP-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanisha Koneru reassigned HADOOP-13710:
---
Assignee: Hanisha Koneru
> Supress CachingGetSpaceUsed from logging interrupted exception stacktrace
> -
>
> Key: HADOOP-13710
> URL: https://issues.apache.org/jira/browse/HADOOP-13710
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Hanisha Koneru
>Priority: Minor
> Labels: supportability
>
> CachingGetSpaceUsed thread is typically interrupted when the node is
> shutdown. Since this is a routine operation, there seems not much value to
> print the stacktrace of an {{InterruptedException}}.
> {quote}
> 2016-10-11 10:02:25,894 WARN fs.CachingGetSpaceUsed
> (CachingGetSpaceUsed.java:run(180)) - Thread Interrupted waiting to refresh
> disk information
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:176)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13708) Fix a few typos in site *.md documents
[ https://issues.apache.org/jira/browse/HADOOP-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569968#comment-15569968 ] Andrew Wang commented on HADOOP-13708: -- Great work here, thanks for sending us a patch. I have a couple small nits, some of which are not related to your changes but things I saw while looking at the diff. Some things I think read better as the original: * "Identifying the audience of an interface helps defining the impact of breaking" * "The interface is for general use by any applications." * "implementing rolling upgrades. It communicates that this interface should" * Prefer we leave the empty Precondition headers in the FileSystem specification * "dest` will match those under `src`, as will the contents do:" Suggestions: * "e.g. In HDFS, FSImage stability can help providing more flexible roll backs." -> "provide more flexible rollback." * "A good example of a limited-private interface is BlockLocations, This is" -> "A good example of a limited-private interface is BlockLocations. This is a" * "have got a coordinated effort with the MR team to release matching releases" -> "coordinate release efforts with the MR team" * "While the caller may expect for as much buffer as possible to be filled" -> "While the caller may expect as much of the buffer as possible to be filled" * "be used as direct replacement for HDFS." -> "be used as direct replacements for HDFS." > Fix a few typos in site *.md documents > -- > > Key: HADOOP-13708 > URL: https://issues.apache.org/jira/browse/HADOOP-13708 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.8.0 >Reporter: Ding Fei >Assignee: Ding Fei >Priority: Minor > Attachments: HADOOP-13708.patch > > > Fix several typos in site *.md documents. > Touched documents listed: > * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/notation.md > * > hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md > * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md > * > hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md > * hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md > * hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13711) Supress CachingGetSpaceUsed from logging interrupted exception stacktrace
[
https://issues.apache.org/jira/browse/HADOOP-13711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanisha Koneru reassigned HADOOP-13711:
---
Assignee: Hanisha Koneru
> Supress CachingGetSpaceUsed from logging interrupted exception stacktrace
> -
>
> Key: HADOOP-13711
> URL: https://issues.apache.org/jira/browse/HADOOP-13711
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Hanisha Koneru
>Priority: Minor
> Labels: supportability
>
> CachingGetSpaceUsed thread is typically interrupted when the node is
> shutdown. Since this is a routine operation, there seems not much value to
> print the stacktrace of an {{InterruptedException}}.
> {quote}
> 2016-10-11 10:02:25,894 WARN fs.CachingGetSpaceUsed
> (CachingGetSpaceUsed.java:run(180)) - Thread Interrupted waiting to refresh
> disk information
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:176)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13560) S3ABlockOutputStream to support huge (many GB) file writes
[
https://issues.apache.org/jira/browse/HADOOP-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569842#comment-15569842
]
Chris Nauroth commented on HADOOP-13560:
The trunk patch looks good to me. I have tested successfully against us-west-2.
However, I just noticed that after I apply the branch-2 patch and build the
distro, every {{hadoop fs}} command accessing S3A prints a deprecation warning:
{code}
16/10/12 13:48:53 INFO Configuration.deprecation: Unsupported option
"fs.s3a.threads.core" will be ignored
{code}
I think that's because the patch added the {{DeprecationDelta}}, but
{{S3AFileSystem#initialize}} still has code that tries to access
{{fs.s3a.threads.core}}.
> S3ABlockOutputStream to support huge (many GB) file writes
> --
>
> Key: HADOOP-13560
> URL: https://issues.apache.org/jira/browse/HADOOP-13560
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13560-branch-2-001.patch,
> HADOOP-13560-branch-2-002.patch, HADOOP-13560-branch-2-003.patch,
> HADOOP-13560-branch-2-004.patch
>
>
> An AWS SDK [issue|https://github.com/aws/aws-sdk-java/issues/367] highlights
> that metadata isn't copied on large copies.
> 1. Add a test to do that large copy/rname and verify that the copy really
> works
> 2. Verify that metadata makes it over.
> Verifying large file rename is important on its own, as it is needed for very
> large commit operations for committers using rename
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13716) Add an Eventually class to for tests to await eventual completion of lambda expressions/test clauses
[
https://issues.apache.org/jira/browse/HADOOP-13716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569830#comment-15569830
]
Hadoop QA commented on HADOOP-13716:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 13 new + 0 unchanged - 0 fixed = 13 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
8s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13716 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832968/HADOOP-13716-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e3d138890d49 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6476934 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10748/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10748/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10748/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add an Eventually class to for tests to await eventual completion of lambda
> expressions/test clauses
>
>
> Key: HADOOP-13716
> URL: https://issues.apache.org/jira/browse/HADOOP-13716
> Project: Hadoop Common
>
[jira] [Assigned] (HADOOP-13715) Add isErasureCoded() API to FileStatus class
[
https://issues.apache.org/jira/browse/HADOOP-13715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang reassigned HADOOP-13715:
Assignee: Wei-Chiu Chuang
> Add isErasureCoded() API to FileStatus class
>
>
> Key: HADOOP-13715
> URL: https://issues.apache.org/jira/browse/HADOOP-13715
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>
> Per the discussion in
> [HDFS-10971|https://issues.apache.org/jira/browse/HDFS-10971?focusedCommentId=15567108=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15567108]
> I would like to add a new API {{isErasureCoded()}} to {{FileStatus}} so that
> tools and downstream applications can tell if it needs to treat a file
> differently.
> Hadoop tools that can benefit from this effort include: distcp and
> teragen/terasort.
> Downstream applications such as flume or hbase may also benefit from it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13669) KMS Server should log exceptions before throwing
[
https://issues.apache.org/jira/browse/HADOOP-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569758#comment-15569758
]
Xiao Chen commented on HADOOP-13669:
{noformat}
findbugs detection: trunk
...
hadoop-common-project/hadoop-kms in trunk has 2 extant Findbugs warnings.
findbugs detection: patch
cd /testptch/hadoop/hadoop-common-project/hadoop-kms
mvn -Dmaven.repo.local=/home/jenkins/yetus-m2/hadoop-trunk-patch-1 -Ptest-patch
-DskipTests test-compile findbugs:findbugs -DskipTests=true >
/testptch/hadoop/patchprocess/patch-findbugs-hadoop-common-project_hadoop-kms.txt
2>&1
Elapsed: 0m 21s
Starting with
/testptch/hadoop/patchprocess/branch-findbugs-hadoop-common-project_hadoop-kms-warnings.xml
Merging
/testptch/hadoop/patchprocess/patch-findbugs-hadoop-common-project_hadoop-kms-warnings.xml
Writing
/testptch/hadoop/patchprocess/combined-findbugs-hadoop-common-project_hadoop-kms.xml
{noformat}
The addendum patch fixed the errors. I think this is ready for commit, [~aw]
could you take a look? Thanks a lot.
> KMS Server should log exceptions before throwing
>
>
> Key: HADOOP-13669
> URL: https://issues.apache.org/jira/browse/HADOOP-13669
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Reporter: Xiao Chen
>Assignee: Suraj Acharya
> Labels: supportability
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13369.2.patch, HADOOP-13369.patch,
> HADOOP-13369.patch.1, HADOOP-13669.addendum.patch, trigger.patch
>
>
> In some recent investigation, it turns out when KMS throws an exception (into
> tomcat), it's not logged anywhere and we can only see the exception message
> from client-side, but not the stacktrace. Logging the stacktrance would help
> debugging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13713) ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive failing intermittently
[
https://issues.apache.org/jira/browse/HADOOP-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569744#comment-15569744
]
Steve Loughran commented on HADOOP-13713:
-
Added HADOOP-13716 for the general class of this, designed to work in Java 7
but better in Java 8. We can use this for the FS contract dependency problems,
look at it in other places too.
> ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive failing intermittently
> -
>
> Key: HADOOP-13713
> URL: https://issues.apache.org/jira/browse/HADOOP-13713
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: s3 ireland
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> intermittent failure of
> {{ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive}} surfacing in
> HADOOP-12774 test run.
> This is a test which came in with HADOOP-12977, one test which deletes all
> children of the root dir, then verifies that they are gone. Although it
> tested happily during development, the sightings of two transient failures
> before it worked implied that it's either got some race condition with
> previous tests and/or maven build, or we are seeing listing inconsistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13716) Add an Eventually class to for tests to await eventual completion of lambda expressions/test clauses
[
https://issues.apache.org/jira/browse/HADOOP-13716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569723#comment-15569723
]
Steve Loughran edited comment on HADOOP-13716 at 10/12/16 8:05 PM:
---
example from a test
{code}
int reps = eventually(
() -> ++count == 4,
50,
() -> 10,
(timeout, ex) -> new Exception("timeout after " + count)
);
{code}
This will evaluate the first closure until it succeeds or the total wait time
exceeds 50; the operation for interval calculation is always 10. Returns the
number of iterations it took.
Another example, evaluate a closure until it succeeds. Here it doesn't; the
exception caught on the final unsuccessful iteration is rethrown.
{code}
@Test
public void testEvalLambdaFailures() throws Throwable {
try {
evaluate(() -> {
throw new IOException("oops");
},
TIMEOUT,
retry);
fail("should not have got here");
} catch (IOException expected) {
assertExceptionContains("oops", expected);
}
}
{code}
As well as being Lambda friendly, the patch is just invoking {{Callable}},
so works with Java & and normal interface/implementation classes; that's how I
plan to initially use it for branch-2 code. I just want to be confident we have
something which will move to lambda-expressions with ease.
Also, I've added a FailFastException, similar to that in S3ATestUtils. If a
closure throws that: no attempt to retry. Code that knows what it is doing can
use that to bail out fast on a condition which will never be met.
I'm looking at doing some S3A work, in particular handling inconsistency
problems in some of the FS contract root tests, though I think someone ought to
be able to migrate {{waitFor()}} code to it when they want better exception
propagation, failure handling or alternative backoff strategies
was (Author: ste...@apache.org):
example from a test
{code}
int reps = eventually(
() -> ++count == 4,
50,
() -> 10,
(timeout, ex) -> new Exception("timeout after " + count)
);
{code}
This will evaluate the first closure until it succeeds or the total wait time
exceeds 50; the operation for interval calculation is always 10. Returns the
number of iterations it took.
Another example, evaluate a closure until it succeeds. Here it doesn't; the e
{code}
@Test
public void testEvalLambdaFailures() throws Throwable {
try {
evaluate(() -> {
throw new IOException("oops");
},
TIMEOUT,
retry);
fail("should not have got here");
} catch (IOException expected) {
assertExceptionContains("oops", expected);
}
}
{code}
As well as being Lambda friendly, the patch is just invoking {{Callable}},
so works with Java & and normal interface/implementation classes; that's how I
plan to initially use it for branch-2 code. I just want to be confident we have
something which will move to lambda-expressions with ease.
I'm looking at doing some S3A work, in particular handling inconsistency
problems in some of the FS contract root tests, though I think someone ought to
be able to migrate {{waitFor()}} code to it when they want better exception
propagation, failure handling or alternative backoff strategies
> Add an Eventually class to for tests to await eventual completion of lambda
> expressions/test clauses
>
>
> Key: HADOOP-13716
> URL: https://issues.apache.org/jira/browse/HADOOP-13716
> Project: Hadoop Common
> Issue Type: New Feature
> Components: test
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13716-001.patch
>
>
> To make our tests robust against timing problems and eventual consistent
> stores, we need to do more spin & wait for state.
> We have some code in {{GenericTestUtils.waitFor}} to await a condition being
> met, but the predicate it calls doesn't throw exceptions, there's no way for
> a probe to throw an exception, and all you get is the eventual "timed out"
> message.
> We can do better, and in closure-ready languages (scala & scalatest, groovy
> and some slider code) we've examples to follow. Some of that work has been
> reimplemented slightly in {{S3ATestUtils.eventually}}
> I propose adding a class in the test tree, {{Eventually}} to be a
> successor/replacement for these.
> # has an eventually/waitfor operation taking a predicate that throws an
> exception
> # has an "evaluate" exception which tries to evaluate an answer until the
> operation stops raising an exception. (again, from scalatest)
> # plugin backoff strategies (from Scalatest; lets you do exponential as well
> as linear)
[jira] [Updated] (HADOOP-13716) Add an Eventually class to for tests to await eventual completion of lambda expressions/test clauses
[
https://issues.apache.org/jira/browse/HADOOP-13716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13716:
Status: Patch Available (was: Open)
> Add an Eventually class to for tests to await eventual completion of lambda
> expressions/test clauses
>
>
> Key: HADOOP-13716
> URL: https://issues.apache.org/jira/browse/HADOOP-13716
> Project: Hadoop Common
> Issue Type: New Feature
> Components: test
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13716-001.patch
>
>
> To make our tests robust against timing problems and eventual consistent
> stores, we need to do more spin & wait for state.
> We have some code in {{GenericTestUtils.waitFor}} to await a condition being
> met, but the predicate it calls doesn't throw exceptions, there's no way for
> a probe to throw an exception, and all you get is the eventual "timed out"
> message.
> We can do better, and in closure-ready languages (scala & scalatest, groovy
> and some slider code) we've examples to follow. Some of that work has been
> reimplemented slightly in {{S3ATestUtils.eventually}}
> I propose adding a class in the test tree, {{Eventually}} to be a
> successor/replacement for these.
> # has an eventually/waitfor operation taking a predicate that throws an
> exception
> # has an "evaluate" exception which tries to evaluate an answer until the
> operation stops raising an exception. (again, from scalatest)
> # plugin backoff strategies (from Scalatest; lets you do exponential as well
> as linear)
> # option of adding a special handler to generate the failure exception (e.g.
> run more detailed diagnostics for the exception text, etc).
> # be Java 8 lambda expression friendly
> # be testable and tested itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13716) Add an Eventually class to for tests to await eventual completion of lambda expressions/test clauses
[
https://issues.apache.org/jira/browse/HADOOP-13716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13716:
Attachment: HADOOP-13716-001.patch
Patch 001.
The production code is Java7+; the tests are of both java7 and java 8
code...for backporting to java 8 you'd just pull those test cases with the word
"lambda" in.
Because this is needed to improve some of the FS contract tests, they should be
a PoC that the API works —I'd be targeting Java 7 there.
> Add an Eventually class to for tests to await eventual completion of lambda
> expressions/test clauses
>
>
> Key: HADOOP-13716
> URL: https://issues.apache.org/jira/browse/HADOOP-13716
> Project: Hadoop Common
> Issue Type: New Feature
> Components: test
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13716-001.patch
>
>
> To make our tests robust against timing problems and eventual consistent
> stores, we need to do more spin & wait for state.
> We have some code in {{GenericTestUtils.waitFor}} to await a condition being
> met, but the predicate it calls doesn't throw exceptions, there's no way for
> a probe to throw an exception, and all you get is the eventual "timed out"
> message.
> We can do better, and in closure-ready languages (scala & scalatest, groovy
> and some slider code) we've examples to follow. Some of that work has been
> reimplemented slightly in {{S3ATestUtils.eventually}}
> I propose adding a class in the test tree, {{Eventually}} to be a
> successor/replacement for these.
> # has an eventually/waitfor operation taking a predicate that throws an
> exception
> # has an "evaluate" exception which tries to evaluate an answer until the
> operation stops raising an exception. (again, from scalatest)
> # plugin backoff strategies (from Scalatest; lets you do exponential as well
> as linear)
> # option of adding a special handler to generate the failure exception (e.g.
> run more detailed diagnostics for the exception text, etc).
> # be Java 8 lambda expression friendly
> # be testable and tested itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13716) Add an Eventually class to for tests to await eventual completion of lambda expressions/test clauses
[
https://issues.apache.org/jira/browse/HADOOP-13716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13716:
Summary: Add an Eventually class to for tests to await eventual completion
of lambda expressions/test clauses (was: Add an Eventually class to await
eventual completion of lambda expressions)
> Add an Eventually class to for tests to await eventual completion of lambda
> expressions/test clauses
>
>
> Key: HADOOP-13716
> URL: https://issues.apache.org/jira/browse/HADOOP-13716
> Project: Hadoop Common
> Issue Type: New Feature
> Components: test
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> To make our tests robust against timing problems and eventual consistent
> stores, we need to do more spin & wait for state.
> We have some code in {{GenericTestUtils.waitFor}} to await a condition being
> met, but the predicate it calls doesn't throw exceptions, there's no way for
> a probe to throw an exception, and all you get is the eventual "timed out"
> message.
> We can do better, and in closure-ready languages (scala & scalatest, groovy
> and some slider code) we've examples to follow. Some of that work has been
> reimplemented slightly in {{S3ATestUtils.eventually}}
> I propose adding a class in the test tree, {{Eventually}} to be a
> successor/replacement for these.
> # has an eventually/waitfor operation taking a predicate that throws an
> exception
> # has an "evaluate" exception which tries to evaluate an answer until the
> operation stops raising an exception. (again, from scalatest)
> # plugin backoff strategies (from Scalatest; lets you do exponential as well
> as linear)
> # option of adding a special handler to generate the failure exception (e.g.
> run more detailed diagnostics for the exception text, etc).
> # be Java 8 lambda expression friendly
> # be testable and tested itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13716) Add an Eventually class to await eventual completion of lambda expressions
[
https://issues.apache.org/jira/browse/HADOOP-13716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reassigned HADOOP-13716:
---
Assignee: Steve Loughran
> Add an Eventually class to await eventual completion of lambda expressions
> --
>
> Key: HADOOP-13716
> URL: https://issues.apache.org/jira/browse/HADOOP-13716
> Project: Hadoop Common
> Issue Type: New Feature
> Components: test
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> To make our tests robust against timing problems and eventual consistent
> stores, we need to do more spin & wait for state.
> We have some code in {{GenericTestUtils.waitFor}} to await a condition being
> met, but the predicate it calls doesn't throw exceptions, there's no way for
> a probe to throw an exception, and all you get is the eventual "timed out"
> message.
> We can do better, and in closure-ready languages (scala & scalatest, groovy
> and some slider code) we've examples to follow. Some of that work has been
> reimplemented slightly in {{S3ATestUtils.eventually}}
> I propose adding a class in the test tree, {{Eventually}} to be a
> successor/replacement for these.
> # has an eventually/waitfor operation taking a predicate that throws an
> exception
> # has an "evaluate" exception which tries to evaluate an answer until the
> operation stops raising an exception. (again, from scalatest)
> # plugin backoff strategies (from Scalatest; lets you do exponential as well
> as linear)
> # option of adding a special handler to generate the failure exception (e.g.
> run more detailed diagnostics for the exception text, etc).
> # be Java 8 lambda expression friendly
> # be testable and tested itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13716) Add an Eventually class to await eventual completion of lambda expressions
[
https://issues.apache.org/jira/browse/HADOOP-13716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569723#comment-15569723
]
Steve Loughran commented on HADOOP-13716:
-
example from a test
{code}
int reps = eventually(
() -> ++count == 4,
50,
() -> 10,
(timeout, ex) -> new Exception("timeout after " + count)
);
{code}
This will evaluate the first closure until it succeeds or the total wait time
exceeds 50; the operation for interval calculation is always 10. Returns the
number of iterations it took.
Another example, evaluate a closure until it succeeds. Here it doesn't; the e
{code}
@Test
public void testEvalLambdaFailures() throws Throwable {
try {
evaluate(() -> {
throw new IOException("oops");
},
TIMEOUT,
retry);
fail("should not have got here");
} catch (IOException expected) {
assertExceptionContains("oops", expected);
}
}
{code}
As well as being Lambda friendly, the patch is just invoking {{Callable}},
so works with Java & and normal interface/implementation classes; that's how I
plan to initially use it for branch-2 code. I just want to be confident we have
something which will move to lambda-expressions with ease.
I'm looking at doing some S3A work, in particular handling inconsistency
problems in some of the FS contract root tests, though I think someone ought to
be able to migrate {{waitFor()}} code to it when they want better exception
propagation, failure handling or alternative backoff strategies
> Add an Eventually class to await eventual completion of lambda expressions
> --
>
> Key: HADOOP-13716
> URL: https://issues.apache.org/jira/browse/HADOOP-13716
> Project: Hadoop Common
> Issue Type: New Feature
> Components: test
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>
> To make our tests robust against timing problems and eventual consistent
> stores, we need to do more spin & wait for state.
> We have some code in {{GenericTestUtils.waitFor}} to await a condition being
> met, but the predicate it calls doesn't throw exceptions, there's no way for
> a probe to throw an exception, and all you get is the eventual "timed out"
> message.
> We can do better, and in closure-ready languages (scala & scalatest, groovy
> and some slider code) we've examples to follow. Some of that work has been
> reimplemented slightly in {{S3ATestUtils.eventually}}
> I propose adding a class in the test tree, {{Eventually}} to be a
> successor/replacement for these.
> # has an eventually/waitfor operation taking a predicate that throws an
> exception
> # has an "evaluate" exception which tries to evaluate an answer until the
> operation stops raising an exception. (again, from scalatest)
> # plugin backoff strategies (from Scalatest; lets you do exponential as well
> as linear)
> # option of adding a special handler to generate the failure exception (e.g.
> run more detailed diagnostics for the exception text, etc).
> # be Java 8 lambda expression friendly
> # be testable and tested itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13716) Add an Eventually class to await eventual completion of lambda expressions
Steve Loughran created HADOOP-13716:
---
Summary: Add an Eventually class to await eventual completion of
lambda expressions
Key: HADOOP-13716
URL: https://issues.apache.org/jira/browse/HADOOP-13716
Project: Hadoop Common
Issue Type: New Feature
Components: test
Affects Versions: 2.8.0
Reporter: Steve Loughran
To make our tests robust against timing problems and eventual consistent
stores, we need to do more spin & wait for state.
We have some code in {{GenericTestUtils.waitFor}} to await a condition being
met, but the predicate it calls doesn't throw exceptions, there's no way for a
probe to throw an exception, and all you get is the eventual "timed out"
message.
We can do better, and in closure-ready languages (scala & scalatest, groovy and
some slider code) we've examples to follow. Some of that work has been
reimplemented slightly in {{S3ATestUtils.eventually}}
I propose adding a class in the test tree, {{Eventually}} to be a
successor/replacement for these.
# has an eventually/waitfor operation taking a predicate that throws an
exception
# has an "evaluate" exception which tries to evaluate an answer until the
operation stops raising an exception. (again, from scalatest)
# plugin backoff strategies (from Scalatest; lets you do exponential as well as
linear)
# option of adding a special handler to generate the failure exception (e.g.
run more detailed diagnostics for the exception text, etc).
# be Java 8 lambda expression friendly
# be testable and tested itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13713) ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive failing intermittently
[
https://issues.apache.org/jira/browse/HADOOP-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569680#comment-15569680
]
Steve Loughran commented on HADOOP-13713:
-
afraid so. Writing a nice java-8 ready thing for eventual probes/evaluation;
with tests
> ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive failing intermittently
> -
>
> Key: HADOOP-13713
> URL: https://issues.apache.org/jira/browse/HADOOP-13713
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: s3 ireland
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> intermittent failure of
> {{ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive}} surfacing in
> HADOOP-12774 test run.
> This is a test which came in with HADOOP-12977, one test which deletes all
> children of the root dir, then verifies that they are gone. Although it
> tested happily during development, the sightings of two transient failures
> before it worked implied that it's either got some race condition with
> previous tests and/or maven build, or we are seeing listing inconsistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13715) Add isErasureCoded() API to FileStatus class
Wei-Chiu Chuang created HADOOP-13715:
Summary: Add isErasureCoded() API to FileStatus class
Key: HADOOP-13715
URL: https://issues.apache.org/jira/browse/HADOOP-13715
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Affects Versions: 3.0.0-alpha1
Reporter: Wei-Chiu Chuang
Per the discussion in
[HDFS-10971|https://issues.apache.org/jira/browse/HDFS-10971?focusedCommentId=15567108=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15567108]
I would like to add a new API {{isErasureCoded()}} to {{FileStatus}} so that
tools and downstream applications can tell if it needs to treat a file
differently.
Hadoop tools that can benefit from this effort include: distcp and
teragen/terasort.
Downstream applications such as flume or hbase may also benefit from it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13669) KMS Server should log exceptions before throwing
[ https://issues.apache.org/jira/browse/HADOOP-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569465#comment-15569465 ] Suraj Acharya commented on HADOOP-13669: The patch looks good to me. Sorry about that [~aw] and [~xiaochen]. > KMS Server should log exceptions before throwing > > > Key: HADOOP-13669 > URL: https://issues.apache.org/jira/browse/HADOOP-13669 > Project: Hadoop Common > Issue Type: Improvement > Components: kms >Reporter: Xiao Chen >Assignee: Suraj Acharya > Labels: supportability > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13369.2.patch, HADOOP-13369.patch, > HADOOP-13369.patch.1, HADOOP-13669.addendum.patch, trigger.patch > > > In some recent investigation, it turns out when KMS throws an exception (into > tomcat), it's not logged anywhere and we can only see the exception message > from client-side, but not the stacktrace. Logging the stacktrance would help > debugging. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13669) KMS Server should log exceptions before throwing
[
https://issues.apache.org/jira/browse/HADOOP-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569434#comment-15569434
]
Hadoop QA commented on HADOOP-13669:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} hadoop-common-project/hadoop-kms in trunk has 2 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
14s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13669 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832943/trigger.patch |
| Optional Tests | asflicense xml compile javac javadoc mvninstall
mvnsite unit findbugs checkstyle |
| uname | Linux dddcca21f9d6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6476934 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10747/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-kms-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10747/testReport/ |
| modules | C: hadoop-common-project/hadoop-kms U:
hadoop-common-project/hadoop-kms |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10747/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> KMS Server should log exceptions before throwing
>
>
> Key:
[jira] [Comment Edited] (HADOOP-13714) Tighten up our compatibility guidelines for Hadoop 3
[ https://issues.apache.org/jira/browse/HADOOP-13714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569408#comment-15569408 ] Karthik Kambatla edited comment on HADOOP-13714 at 10/12/16 5:55 PM: - Created this JIRA so we don't miss it, and assigning to myself. Will not be able to get to this this month. Please feel free to pick this up if you are interested an have cycles to work on this. was (Author: kasha): Created this JIRA so we don't miss it, and assigning to myself. Will not be able to get to this in the next month. Please feel free to pick this up if you are interested an have cycles to work on this. > Tighten up our compatibility guidelines for Hadoop 3 > > > Key: HADOOP-13714 > URL: https://issues.apache.org/jira/browse/HADOOP-13714 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.3 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Blocker > > Our current compatibility guidelines are incomplete and loose. For many > categories, we do not have a policy. It would be nice to actually define > those policies so our users know what to expect and the developers know what > releases to target their changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13714) Tighten up our compatibility guidelines for Hadoop 3
[ https://issues.apache.org/jira/browse/HADOOP-13714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569408#comment-15569408 ] Karthik Kambatla edited comment on HADOOP-13714 at 10/12/16 5:55 PM: - Created this JIRA so we don't miss it, and assigning to myself. Will not be able to get to this, this month. Please feel free to pick this up if you are interested an have cycles to work on this. was (Author: kasha): Created this JIRA so we don't miss it, and assigning to myself. Will not be able to get to this this month. Please feel free to pick this up if you are interested an have cycles to work on this. > Tighten up our compatibility guidelines for Hadoop 3 > > > Key: HADOOP-13714 > URL: https://issues.apache.org/jira/browse/HADOOP-13714 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.3 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Blocker > > Our current compatibility guidelines are incomplete and loose. For many > categories, we do not have a policy. It would be nice to actually define > those policies so our users know what to expect and the developers know what > releases to target their changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13714) Tighten up our compatibility guidelines for Hadoop 3
[ https://issues.apache.org/jira/browse/HADOOP-13714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569408#comment-15569408 ] Karthik Kambatla commented on HADOOP-13714: --- Created this JIRA so we don't miss it, and assigning to myself. Will not be able to get to this in the next month. Please feel free to pick this up if you are interested an have cycles to work on this. > Tighten up our compatibility guidelines for Hadoop 3 > > > Key: HADOOP-13714 > URL: https://issues.apache.org/jira/browse/HADOOP-13714 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.3 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Blocker > > Our current compatibility guidelines are incomplete and loose. For many > categories, we do not have a policy. It would be nice to actually define > those policies so our users know what to expect and the developers know what > releases to target their changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13714) Tighten up our compatibility guidelines for Hadoop 3
Karthik Kambatla created HADOOP-13714: - Summary: Tighten up our compatibility guidelines for Hadoop 3 Key: HADOOP-13714 URL: https://issues.apache.org/jira/browse/HADOOP-13714 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: 2.7.3 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Blocker Our current compatibility guidelines are incomplete and loose. For many categories, we do not have a policy. It would be nice to actually define those policies so our users know what to expect and the developers know what releases to target their changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13669) KMS Server should log exceptions before throwing
[ https://issues.apache.org/jira/browse/HADOOP-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13669: --- Attachment: trigger.patch Let's trigger the findbugs check to be sure... > KMS Server should log exceptions before throwing > > > Key: HADOOP-13669 > URL: https://issues.apache.org/jira/browse/HADOOP-13669 > Project: Hadoop Common > Issue Type: Improvement > Components: kms >Reporter: Xiao Chen >Assignee: Suraj Acharya > Labels: supportability > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13369.2.patch, HADOOP-13369.patch, > HADOOP-13369.patch.1, HADOOP-13669.addendum.patch, trigger.patch > > > In some recent investigation, it turns out when KMS throws an exception (into > tomcat), it's not logged anywhere and we can only see the exception message > from client-side, but not the stacktrace. Logging the stacktrance would help > debugging. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits
[
https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569128#comment-15569128
]
Eric Badger edited comment on HADOOP-13709 at 10/12/16 5:31 PM:
The TestShell#testShellCommandTimerLeak failure in the precommit seems to be
related to this patch. Locally it failed due to timeout 8/500 times with the
patch, but 0/500 times without the patch. The failure is happening when the
code gets blocked calling {{inReader.close()}}. Right above this code there is
a comment about how there was a JDK 7 issue that caused a race with trying to
close the stream, but that was supposedly fixed by the addition of
{{synchronized (stdout)}}.
was (Author: ebadger):
The TestShell#testShellCommandTimerLeak failure in the precommit seems to be
related to this patch. It fails with the patch due to timeout ~1/30 times, but
0/100 without the patch. The failure is happening when the code gets blocked
calling {{inReader.close()}}. Right above this code there is a comment about
how there was a JDK 7 issue that caused a race with trying to close the stream,
but that was supposedly fixed by the addition of {{synchronized (stdout)}}.
> Clean up subprocesses spawned by Shell.java:runCommand when the shell process
> exits
> ---
>
> Key: HADOOP-13709
> URL: https://issues.apache.org/jira/browse/HADOOP-13709
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HADOOP-13709.001.patch
>
>
> The runCommand code in Shell.java can get into a situation where it will
> ignore InterruptedExceptions and refuse to shutdown due to being in I/O
> waiting for the return value of the subprocess that was spawned. We need to
> allow for the subprocess to be interrupted and killed when the shell process
> gets killed. Currently the JVM will shutdown and all of the subprocesses will
> be orphaned and not killed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13669) KMS Server should log exceptions before throwing
[
https://issues.apache.org/jira/browse/HADOOP-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569329#comment-15569329
]
Hadoop QA commented on HADOOP-13669:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13669 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832941/HADOOP-13669.addendum.patch
|
| Optional Tests | asflicense xml |
| uname | Linux 9e48ebf421df 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6476934 |
| modules | C: hadoop-common-project/hadoop-kms U:
hadoop-common-project/hadoop-kms |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10746/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> KMS Server should log exceptions before throwing
>
>
> Key: HADOOP-13669
> URL: https://issues.apache.org/jira/browse/HADOOP-13669
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Reporter: Xiao Chen
>Assignee: Suraj Acharya
> Labels: supportability
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13369.2.patch, HADOOP-13369.patch,
> HADOOP-13369.patch.1, HADOOP-13669.addendum.patch
>
>
> In some recent investigation, it turns out when KMS throws an exception (into
> tomcat), it's not logged anywhere and we can only see the exception message
> from client-side, but not the stacktrace. Logging the stacktrance would help
> debugging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13669) KMS Server should log exceptions before throwing
[ https://issues.apache.org/jira/browse/HADOOP-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13669: --- Status: Patch Available (was: Reopened) > KMS Server should log exceptions before throwing > > > Key: HADOOP-13669 > URL: https://issues.apache.org/jira/browse/HADOOP-13669 > Project: Hadoop Common > Issue Type: Improvement > Components: kms >Reporter: Xiao Chen >Assignee: Suraj Acharya > Labels: supportability > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13369.2.patch, HADOOP-13369.patch, > HADOOP-13369.patch.1, HADOOP-13669.addendum.patch > > > In some recent investigation, it turns out when KMS throws an exception (into > tomcat), it's not logged anywhere and we can only see the exception message > from client-side, but not the stacktrace. Logging the stacktrance would help > debugging. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13669) KMS Server should log exceptions before throwing
[ https://issues.apache.org/jira/browse/HADOOP-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13669: --- Attachment: HADOOP-13669.addendum.patch Thanks [~aw] for reporting this. Sorry for breaking things, I wasn't aware findbug reports have such a severe consequence... Attaching the addendum. [~sacharya], could you take a look? > KMS Server should log exceptions before throwing > > > Key: HADOOP-13669 > URL: https://issues.apache.org/jira/browse/HADOOP-13669 > Project: Hadoop Common > Issue Type: Improvement > Components: kms >Reporter: Xiao Chen >Assignee: Suraj Acharya > Labels: supportability > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13369.2.patch, HADOOP-13369.patch, > HADOOP-13369.patch.1, HADOOP-13669.addendum.patch > > > In some recent investigation, it turns out when KMS throws an exception (into > tomcat), it's not logged anywhere and we can only see the exception message > from client-side, but not the stacktrace. Logging the stacktrance would help > debugging. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13309) Document S3A known limitations in file ownership and permission model.
[ https://issues.apache.org/jira/browse/HADOOP-13309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569299#comment-15569299 ] Chris Nauroth commented on HADOOP-13309: That was a pre-commit run that I manually submitted at builds.apache.org. The whitespace warnings are not relevant. I'm going to wait until HADOOP-12774 gets committed first before I commit this one. > Document S3A known limitations in file ownership and permission model. > -- > > Key: HADOOP-13309 > URL: https://issues.apache.org/jira/browse/HADOOP-13309 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Minor > > S3A does not match the implementation of HDFS in its handling of file > ownership and permissions. Fundamental S3 limitations prevent it. This is a > frequent source of confusion for end users. This issue proposes to document > these known limitations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13309) Document S3A known limitations in file ownership and permission model.
[
https://issues.apache.org/jira/browse/HADOOP-13309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569287#comment-15569287
]
Hadoop QA commented on HADOOP-13309:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
33s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 47 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HADOOP-13309 |
| GITHUB PR | https://github.com/apache/hadoop/pull/138 |
| Optional Tests | asflicense mvnsite |
| uname | Linux 5f50f5a28fa8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / e341e51 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10745/artifact/patchprocess/whitespace-eol.txt
|
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10745/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Document S3A known limitations in file ownership and permission model.
> --
>
> Key: HADOOP-13309
> URL: https://issues.apache.org/jira/browse/HADOOP-13309
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
>
> S3A does not match the implementation of HDFS in its handling of file
> ownership and permissions. Fundamental S3 limitations prevent it. This is a
> frequent source of confusion for end users. This issue proposes to document
> these known limitations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13713) ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive failing intermittently
[
https://issues.apache.org/jira/browse/HADOOP-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569289#comment-15569289
]
Chris Nauroth commented on HADOOP-13713:
If we suspect eventual consistency of listings is the root cause, then perhaps
this test needs something similar to what I did for
{{testListEmptyRootDirectory}} in HADOOP-13113 or {{S3ATestUtils#eventually}}.
> ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive failing intermittently
> -
>
> Key: HADOOP-13713
> URL: https://issues.apache.org/jira/browse/HADOOP-13713
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: s3 ireland
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> intermittent failure of
> {{ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive}} surfacing in
> HADOOP-12774 test run.
> This is a test which came in with HADOOP-12977, one test which deletes all
> children of the root dir, then verifies that they are gone. Although it
> tested happily during development, the sightings of two transient failures
> before it worked implied that it's either got some race condition with
> previous tests and/or maven build, or we are seeing listing inconsistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13560) S3ABlockOutputStream to support huge (many GB) file writes
[
https://issues.apache.org/jira/browse/HADOOP-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569243#comment-15569243
]
Chris Nauroth commented on HADOOP-13560:
I see now. I figured the JavaDoc could just link directly to the constant
brought in via {{import static}}, but it appears JavaDoc doesn't support that.
Please disregard my comment, and I'll look at the trunk patch.
> S3ABlockOutputStream to support huge (many GB) file writes
> --
>
> Key: HADOOP-13560
> URL: https://issues.apache.org/jira/browse/HADOOP-13560
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13560-branch-2-001.patch,
> HADOOP-13560-branch-2-002.patch, HADOOP-13560-branch-2-003.patch,
> HADOOP-13560-branch-2-004.patch
>
>
> An AWS SDK [issue|https://github.com/aws/aws-sdk-java/issues/367] highlights
> that metadata isn't copied on large copies.
> 1. Add a test to do that large copy/rname and verify that the copy really
> works
> 2. Verify that metadata makes it over.
> Verifying large file rename is important on its own, as it is needed for very
> large commit operations for committers using rename
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13669) KMS Server should log exceptions before throwing
[ https://issues.apache.org/jira/browse/HADOOP-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer reopened HADOOP-13669: --- bq. Findbugs warnings are false alarm, since the exception is thrown. ... bq. Committed this to trunk, branch-2 and branch-2.8. Re-opening, because this patch needs an addendum. Please don't commit patches that throw findbugs errors. If you are sure this is a false alarm, an exception must get added to findbugs-exclude.xml. Until that happens, the release, the nightlies, and all other patches against hadoop-kms are going to fail. > KMS Server should log exceptions before throwing > > > Key: HADOOP-13669 > URL: https://issues.apache.org/jira/browse/HADOOP-13669 > Project: Hadoop Common > Issue Type: Improvement > Components: kms >Reporter: Xiao Chen >Assignee: Suraj Acharya > Labels: supportability > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13369.2.patch, HADOOP-13369.patch, > HADOOP-13369.patch.1 > > > In some recent investigation, it turns out when KMS throws an exception (into > tomcat), it's not logged anywhere and we can only see the exception message > from client-side, but not the stacktrace. Logging the stacktrance would help > debugging. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits
[
https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569128#comment-15569128
]
Eric Badger commented on HADOOP-13709:
--
The TestShell#testShellCommandTimerLeak failure in the precommit seems to be
related to this patch. It fails with the patch due to timeout ~1/30 times, but
0/100 without the patch. The failure is happening when the code gets blocked
calling {{inReader.close()}}. Right above this code there is a comment about
how there was a JDK 7 issue that caused a race with trying to close the stream,
but that was supposedly fixed by the addition of {{synchronized (stdout)}}.
> Clean up subprocesses spawned by Shell.java:runCommand when the shell process
> exits
> ---
>
> Key: HADOOP-13709
> URL: https://issues.apache.org/jira/browse/HADOOP-13709
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HADOOP-13709.001.patch
>
>
> The runCommand code in Shell.java can get into a situation where it will
> ignore InterruptedExceptions and refuse to shutdown due to being in I/O
> waiting for the return value of the subprocess that was spawned. We need to
> allow for the subprocess to be interrupted and killed when the shell process
> gets killed. Currently the JVM will shutdown and all of the subprocesses will
> be orphaned and not killed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13659) Upgrade jaxb-api version
[ https://issues.apache.org/jira/browse/HADOOP-13659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569085#comment-15569085 ] Sean Mackrory commented on HADOOP-13659: This is primarily used in the YARN web UI - I've also done some manual testing of that to make sure nothing seems broken. Unit test runs continue to give virtually identical results to before this change (I say "virtually" because of transient failures I both seen and not seen both with and without this change - none of them appear related to this change anyway). So I'd like to move ahead with this update... > Upgrade jaxb-api version > > > Key: HADOOP-13659 > URL: https://issues.apache.org/jira/browse/HADOOP-13659 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha2 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HADOOP-13659.001.patch > > > We're currently pulling in version 2.2.2 - I think we should upgrade to the > latest 2.2.12. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13703) S3ABlockOutputStream to pass Yetus & Jenkins
[ https://issues.apache.org/jira/browse/HADOOP-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13703: Status: Open (was: Patch Available) > S3ABlockOutputStream to pass Yetus & Jenkins > > > Key: HADOOP-13703 > URL: https://issues.apache.org/jira/browse/HADOOP-13703 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13560-012.patch, HADOOP-13560-branch-2-010.patch, > HADOOP-13560-branch-2-011.patch > > > The HADOOP-13560 patches and PR has got yetus confused. This patch is purely > to do test runs. > h1. All discourse must continue to take place in HADOOP-13560 and/or the Pull > Request. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13703) S3ABlockOutputStream to pass Yetus & Jenkins
[ https://issues.apache.org/jira/browse/HADOOP-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13703: Status: Patch Available (was: Open) > S3ABlockOutputStream to pass Yetus & Jenkins > > > Key: HADOOP-13703 > URL: https://issues.apache.org/jira/browse/HADOOP-13703 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13560-012.patch, HADOOP-13560-branch-2-010.patch, > HADOOP-13560-branch-2-011.patch > > > The HADOOP-13560 patches and PR has got yetus confused. This patch is purely > to do test runs. > h1. All discourse must continue to take place in HADOOP-13560 and/or the Pull > Request. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13703) S3ABlockOutputStream to pass Yetus & Jenkins
[ https://issues.apache.org/jira/browse/HADOOP-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13703: Attachment: HADOOP-13560-012.patch Patch 012: patch 011 against trunk > S3ABlockOutputStream to pass Yetus & Jenkins > > > Key: HADOOP-13703 > URL: https://issues.apache.org/jira/browse/HADOOP-13703 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13560-012.patch, HADOOP-13560-branch-2-010.patch, > HADOOP-13560-branch-2-011.patch > > > The HADOOP-13560 patches and PR has got yetus confused. This patch is purely > to do test runs. > h1. All discourse must continue to take place in HADOOP-13560 and/or the Pull > Request. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13686) Adding additional unit test for Trash (I)
[
https://issues.apache.org/jira/browse/HADOOP-13686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568945#comment-15568945
]
Weiwei Yang commented on HADOOP-13686:
--
Hello [~xyao]
# I have fixed UT failure in v8 patch, it was because the test gets an old
FileSystem instance from cache. So I added {{FileSystem.closeAll()}} to fix
this problem. It will ensure each test gets a new instance, it should be stable
now.
# The code change in TestHDFSTrash was added by mistake, I have removed, sorry
about that :).
# The failure TestHttpServerLifecycle in latest jenkins run was not related to
this patch, it is a known issue and tracked by HADOOP-13471.
Please help to review this patch and let know if you have further comments.
Thank you
> Adding additional unit test for Trash (I)
> -
>
> Key: HADOOP-13686
> URL: https://issues.apache.org/jira/browse/HADOOP-13686
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Xiaoyu Yao
>Assignee: Weiwei Yang
> Attachments: HADOOP-13686.01.patch, HADOOP-13686.02.patch,
> HADOOP-13686.03.patch, HADOOP-13686.04.patch, HADOOP-13686.05.patch,
> HADOOP-13686.06.patch
>
>
> This ticket is opened to track adding the forllowing unit test in
> hadoop-common.
> #test users can delete their own trash directory
> #test users can delete an empty directory and the directory is moved to trash
> #test fs.trash.interval with invalid values such as 0 or negative
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13686) Adding additional unit test for Trash (I)
[
https://issues.apache.org/jira/browse/HADOOP-13686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568893#comment-15568893
]
Hadoop QA commented on HADOOP-13686:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 76 unchanged - 1 fixed = 76 total (was 77) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 1s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.http.TestHttpServerLifecycle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13686 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832885/HADOOP-13686.06.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f1648a2ae91f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6476934 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10743/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10743/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10743/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Adding additional unit test for Trash (I)
> -
>
> Key: HADOOP-13686
> URL: https://issues.apache.org/jira/browse/HADOOP-13686
> Project: Hadoop Common
> Issue Type: Test
>
[jira] [Commented] (HADOOP-13560) S3ABlockOutputStream to support huge (many GB) file writes
[ https://issues.apache.org/jira/browse/HADOOP-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568882#comment-15568882 ] Steve Loughran commented on HADOOP-13560: - that unused import is needed; its one of those cases where findbugs is wrong. I'm referring to the import in the javadocs; remove it and the cross reference doesn't apply. So: needed. I'll do the trunk branch though. > S3ABlockOutputStream to support huge (many GB) file writes > -- > > Key: HADOOP-13560 > URL: https://issues.apache.org/jira/browse/HADOOP-13560 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13560-branch-2-001.patch, > HADOOP-13560-branch-2-002.patch, HADOOP-13560-branch-2-003.patch, > HADOOP-13560-branch-2-004.patch > > > An AWS SDK [issue|https://github.com/aws/aws-sdk-java/issues/367] highlights > that metadata isn't copied on large copies. > 1. Add a test to do that large copy/rname and verify that the copy really > works > 2. Verify that metadata makes it over. > Verifying large file rename is important on its own, as it is needed for very > large commit operations for committers using rename -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13713) ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive failing intermittently
[
https://issues.apache.org/jira/browse/HADOOP-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568876#comment-15568876
]
Steve Loughran commented on HADOOP-13713:
-
Initial failure at end of a parallel test run
{code}
Failed tests:
ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive:97->Assert.fail:88
Deletion of child entries failed, still have2
s3a://stevel-ireland/fork-7
s3a://stevel-ireland/fork-8
{code}
immediate rerun of test only, with the parallell test profile still enabled
{code}
mvn -T 1C integration-test -Dtest=moo -Dparallel-tests -DtestsThreadCount=8
-Dit.test=ITestS3AContractRootDir
{code}
outcome
{code}
testRmEmptyRootDirNonRecursive(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)
Time elapsed: 0.863 sec <<< FAILURE!
java.lang.AssertionError: Deletion of child entries failed, still have1
s3a://stevel-ireland/fork-1
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive(AbstractContractRootDirectoryTest.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
Next rerun: all well
> ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive failing intermittently
> -
>
> Key: HADOOP-13713
> URL: https://issues.apache.org/jira/browse/HADOOP-13713
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: s3 ireland
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> intermittent failure of
> {{ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive}} surfacing in
> HADOOP-12774 test run.
> This is a test which came in with HADOOP-12977, one test which deletes all
> children of the root dir, then verifies that they are gone. Although it
> tested happily during development, the sightings of two transient failures
> before it worked implied that it's either got some race condition with
> previous tests and/or maven build, or we are seeing listing inconsistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13707) If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot be accessed
[
https://issues.apache.org/jira/browse/HADOOP-13707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568870#comment-15568870
]
Yuanbo Liu commented on HADOOP-13707:
-
Adding SPENGO filter belongs to enabling SPENGO step.
> If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot
> be accessed
> -
>
> Key: HADOOP-13707
> URL: https://issues.apache.org/jira/browse/HADOOP-13707
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yuanbo Liu
>Assignee: Yuanbo Liu
> Labels: security
> Attachments: HADOOP-13707.001.patch, HADOOP-13707.002.patch,
> HADOOP-13707.003.patch
>
>
> In {{HttpServer2#hasAdministratorAccess}}, it uses
> `hadoop.security.authorization` to detect whether HTTP is authenticated.
> It's not correct, because enabling Kerberos and HTTP SPNEGO are two steps. If
> Kerberos is enabled while HTTP SPNEGO is not, some links cannot be accessed,
> such as "/logs", and it will return error message as below:
> {quote}
> HTTP ERROR 403
> Problem accessing /logs/. Reason:
> User dr.who is unauthorized to access this page.
> {quote}
> We should make sure {{HttpServletRequest#getAuthType}} is not null before we
> invoke {{HttpServer2#hasAdministratorAccess}}.
> {{getAuthType}} means to get the authorization scheme of this request
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13713) ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive failing intermittently
Steve Loughran created HADOOP-13713:
---
Summary: ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive
failing intermittently
Key: HADOOP-13713
URL: https://issues.apache.org/jira/browse/HADOOP-13713
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 2.8.0
Environment: s3 ireland
Reporter: Steve Loughran
Assignee: Steve Loughran
intermittent failure of
{{ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive}} surfacing in
HADOOP-12774 test run.
This is a test which came in with HADOOP-12977, one test which deletes all
children of the root dir, then verifies that they are gone. Although it tested
happily during development, the sightings of two transient failures before it
worked implied that it's either got some race condition with previous tests
and/or maven build, or we are seeing listing inconsistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12774) s3a should use UGI.getCurrentUser.getShortname() for username
[
https://issues.apache.org/jira/browse/HADOOP-12774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568851#comment-15568851
]
Steve Loughran commented on HADOOP-12774:
-
Patch 003: Address Chris's comments about using parent class fields. Also,
given that the first contructor for S3AFileStatus is only for directories, the
isdir parameter is always true, hence superflous. Cut and add javadocs for both
constructors.
Tested: s3a ireland, all well except for some intermittent consistency failures
in {{ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive}}. need to file a
JIRA there
> s3a should use UGI.getCurrentUser.getShortname() for username
> -
>
> Key: HADOOP-12774
> URL: https://issues.apache.org/jira/browse/HADOOP-12774
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> S3a users {{System.getProperty("user.name")}} to get the username for the
> homedir. This is wrong, as it doesn't work on a YARN app where the identity
> is set by HADOOP_USER_NAME, or in a doAs clause.
> Obviously, {{UGI.getCurrentUser.getShortname()}} provides that name,
> everywhere.
> This is a simple change in the source, though testing is harder ... probably
> best to try in a doAs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12774) s3a should use UGI.getCurrentUser.getShortname() for username
[
https://issues.apache.org/jira/browse/HADOOP-12774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-12774:
Status: Open (was: Patch Available)
> s3a should use UGI.getCurrentUser.getShortname() for username
> -
>
> Key: HADOOP-12774
> URL: https://issues.apache.org/jira/browse/HADOOP-12774
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> S3a users {{System.getProperty("user.name")}} to get the username for the
> homedir. This is wrong, as it doesn't work on a YARN app where the identity
> is set by HADOOP_USER_NAME, or in a doAs clause.
> Obviously, {{UGI.getCurrentUser.getShortname()}} provides that name,
> everywhere.
> This is a simple change in the source, though testing is harder ... probably
> best to try in a doAs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12774) s3a should use UGI.getCurrentUser.getShortname() for username
[
https://issues.apache.org/jira/browse/HADOOP-12774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-12774:
Target Version/s: 2.8.0
Status: Patch Available (was: Open)
> s3a should use UGI.getCurrentUser.getShortname() for username
> -
>
> Key: HADOOP-12774
> URL: https://issues.apache.org/jira/browse/HADOOP-12774
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> S3a users {{System.getProperty("user.name")}} to get the username for the
> homedir. This is wrong, as it doesn't work on a YARN app where the identity
> is set by HADOOP_USER_NAME, or in a doAs clause.
> Obviously, {{UGI.getCurrentUser.getShortname()}} provides that name,
> everywhere.
> This is a simple change in the source, though testing is harder ... probably
> best to try in a doAs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13707) If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot be accessed
[
https://issues.apache.org/jira/browse/HADOOP-13707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568753#comment-15568753
]
Yuanbo Liu commented on HADOOP-13707:
-
[~jojochuang] Thanks for your comments.
{quote}
I feel like a correct approach is to add a SPENGO filter...
{quote}
Yes you're right, actually I'm ready to add a SPENGO filter with delegation
feature in HADOOP-13119. But as I said, enabling Kerberos and SPENGO are two
steps. If users enable Kerberos without SPENGO, that means the http sever of
the cluster is in non-security environment. In this situation, static user's
authorization shouldn't be checked.
In the very first installation of Hadoop, http sever is also in non-security
environment without any authorization check. So I think the behavior here
should be consistent and "dr.who" issue should be avoid.
Thanks again for your comments, looking forward to your response. :)
> If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot
> be accessed
> -
>
> Key: HADOOP-13707
> URL: https://issues.apache.org/jira/browse/HADOOP-13707
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yuanbo Liu
>Assignee: Yuanbo Liu
> Labels: security
> Attachments: HADOOP-13707.001.patch, HADOOP-13707.002.patch,
> HADOOP-13707.003.patch
>
>
> In {{HttpServer2#hasAdministratorAccess}}, it uses
> `hadoop.security.authorization` to detect whether HTTP is authenticated.
> It's not correct, because enabling Kerberos and HTTP SPNEGO are two steps. If
> Kerberos is enabled while HTTP SPNEGO is not, some links cannot be accessed,
> such as "/logs", and it will return error message as below:
> {quote}
> HTTP ERROR 403
> Problem accessing /logs/. Reason:
> User dr.who is unauthorized to access this page.
> {quote}
> We should make sure {{HttpServletRequest#getAuthType}} is not null before we
> invoke {{HttpServer2#hasAdministratorAccess}}.
> {{getAuthType}} means to get the authorization scheme of this request
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13686) Adding additional unit test for Trash (I)
[ https://issues.apache.org/jira/browse/HADOOP-13686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HADOOP-13686: - Attachment: HADOOP-13686.06.patch > Adding additional unit test for Trash (I) > - > > Key: HADOOP-13686 > URL: https://issues.apache.org/jira/browse/HADOOP-13686 > Project: Hadoop Common > Issue Type: Test >Reporter: Xiaoyu Yao >Assignee: Weiwei Yang > Attachments: HADOOP-13686.01.patch, HADOOP-13686.02.patch, > HADOOP-13686.03.patch, HADOOP-13686.04.patch, HADOOP-13686.05.patch, > HADOOP-13686.06.patch > > > This ticket is opened to track adding the forllowing unit test in > hadoop-common. > #test users can delete their own trash directory > #test users can delete an empty directory and the directory is moved to trash > #test fs.trash.interval with invalid values such as 0 or negative -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13309) Document S3A known limitations in file ownership and permission model.
[ https://issues.apache.org/jira/browse/HADOOP-13309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568598#comment-15568598 ] Steve Loughran commented on HADOOP-13309: - reviewed text, LGTM, +1 Yetus hasn't looked at this —but being an .md only patch, there's not much for it to review > Document S3A known limitations in file ownership and permission model. > -- > > Key: HADOOP-13309 > URL: https://issues.apache.org/jira/browse/HADOOP-13309 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Minor > > S3A does not match the implementation of HDFS in its handling of file > ownership and permissions. Fundamental S3 limitations prevent it. This is a > frequent source of confusion for end users. This issue proposes to document > these known limitations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13707) If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot be accessed
[
https://issues.apache.org/jira/browse/HADOOP-13707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568512#comment-15568512
]
Hadoop QA commented on HADOOP-13707:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
11s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 135 unchanged - 0 fixed = 136 total (was 135) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
8s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13707 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832867/HADOOP-13707.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 867e043ecce8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6476934 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10742/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10742/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10742/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot
> be accessed
> -
>
> Key: HADOOP-13707
> URL: https://issues.apache.org/jira/browse/HADOOP-13707
> Project: Hadoop Common
> Issue Type:
[jira] [Commented] (HADOOP-13707) If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot be accessed
[
https://issues.apache.org/jira/browse/HADOOP-13707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568498#comment-15568498
]
Wei-Chiu Chuang commented on HADOOP-13707:
--
[~yuanbo] thanks for working in a patch, however I am not sure if this the
right approach. Like what Allen said, logs are not supposed to be seen by
non-admin if the cluster is Kerberized. I feel like a correct approach is to
add a SPENGO filter for /logs so that it is accessible for Kerberos users just
like /jmx and /logLevel. Does that make sense?
> If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot
> be accessed
> -
>
> Key: HADOOP-13707
> URL: https://issues.apache.org/jira/browse/HADOOP-13707
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yuanbo Liu
>Assignee: Yuanbo Liu
> Labels: security
> Attachments: HADOOP-13707.001.patch, HADOOP-13707.002.patch,
> HADOOP-13707.003.patch
>
>
> In {{HttpServer2#hasAdministratorAccess}}, it uses
> `hadoop.security.authorization` to detect whether HTTP is authenticated.
> It's not correct, because enabling Kerberos and HTTP SPNEGO are two steps. If
> Kerberos is enabled while HTTP SPNEGO is not, some links cannot be accessed,
> such as "/logs", and it will return error message as below:
> {quote}
> HTTP ERROR 403
> Problem accessing /logs/. Reason:
> User dr.who is unauthorized to access this page.
> {quote}
> We should make sure {{HttpServletRequest#getAuthType}} is not null before we
> invoke {{HttpServer2#hasAdministratorAccess}}.
> {{getAuthType}} means to get the authorization scheme of this request
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
