[jira] [Updated] (HADOOP-15234) NPE when initializing KMSWebApp
[
https://issues.apache.org/jira/browse/HADOOP-15234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fang zhenyi updated HADOOP-15234:
-
Attachment: HADOOP-15234.001.patch
> NPE when initializing KMSWebApp
> ---
>
> Key: HADOOP-15234
> URL: https://issues.apache.org/jira/browse/HADOOP-15234
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Xiao Chen
>Assignee: fang zhenyi
>Priority: Major
> Attachments: HADOOP-15234.001.patch
>
>
> During KMS startup, if the {{keyProvider}} is null, it will NPE inside
> KeyProviderExtension.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.crypto.key.KeyProviderExtension.(KeyProviderExtension.java:43)
> at
> org.apache.hadoop.crypto.key.CachingKeyProvider.(CachingKeyProvider.java:93)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSWebApp.contextInitialized(KMSWebApp.java:170)
> {noformat}
> We're investigating the exact scenario that could lead to this, but the NPE
> and log around it can be improved.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15234) NPE when initializing KMSWebApp
[
https://issues.apache.org/jira/browse/HADOOP-15234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fang zhenyi updated HADOOP-15234:
-
Status: Patch Available (was: Open)
> NPE when initializing KMSWebApp
> ---
>
> Key: HADOOP-15234
> URL: https://issues.apache.org/jira/browse/HADOOP-15234
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Xiao Chen
>Assignee: fang zhenyi
>Priority: Major
> Attachments: HADOOP-15234.001.patch
>
>
> During KMS startup, if the {{keyProvider}} is null, it will NPE inside
> KeyProviderExtension.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.crypto.key.KeyProviderExtension.(KeyProviderExtension.java:43)
> at
> org.apache.hadoop.crypto.key.CachingKeyProvider.(CachingKeyProvider.java:93)
> at
> org.apache.hadoop.crypto.key.kms.server.KMSWebApp.contextInitialized(KMSWebApp.java:170)
> {noformat}
> We're investigating the exact scenario that could lead to this, but the NPE
> and log around it can be improved.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14588) Fix almost hundreds of typos --some of them are exposed to user.
[ https://issues.apache.org/jira/browse/HADOOP-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379731#comment-16379731 ] LiXin Ge commented on HADOOP-14588: --- [~shahrs87] thanks for your kindly review. [~qwertymaniac], [~ste...@apache.org] could you please have a look at this? I have rebased the patch and it seems affect too many modules in Hadoop that the jenkins get OOM after 8 hours running. Maybe I should divide it into several Jira? > Fix almost hundreds of typos --some of them are exposed to user. > > > Key: HADOOP-14588 > URL: https://issues.apache.org/jira/browse/HADOOP-14588 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LiXin Ge >Assignee: LiXin Ge >Priority: Major > Attachments: HADOOP-14588.001.patch, HADOOP-14588.002.patch > > > There are almost hundreds of wrong spelling, some wrong spelling in document > file may expose to user. Fix them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14588) Fix almost hundreds of typos --some of them are exposed to user.
[ https://issues.apache.org/jira/browse/HADOOP-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiXin Ge updated HADOOP-14588: -- Summary: Fix almost hundreds of typos --some of them are exposed to user. (was: Fix almost hundreds of wrong spelling--some of them are exposed to user.) > Fix almost hundreds of typos --some of them are exposed to user. > > > Key: HADOOP-14588 > URL: https://issues.apache.org/jira/browse/HADOOP-14588 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LiXin Ge >Assignee: LiXin Ge >Priority: Major > Attachments: HADOOP-14588.001.patch, HADOOP-14588.002.patch > > > There are almost hundreds of wrong spelling, some wrong spelling in document > file may expose to user. Fix them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14588) Fix almost hundreds of wrong spelling--some of them are exposed to user.
[ https://issues.apache.org/jira/browse/HADOOP-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiXin Ge updated HADOOP-14588: -- Status: Patch Available (was: Open) > Fix almost hundreds of wrong spelling--some of them are exposed to user. > > > Key: HADOOP-14588 > URL: https://issues.apache.org/jira/browse/HADOOP-14588 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LiXin Ge >Assignee: LiXin Ge >Priority: Major > Attachments: HADOOP-14588.001.patch, HADOOP-14588.002.patch > > > There are almost hundreds of wrong spelling, some wrong spelling in document > file may expose to user. Fix them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14588) Fix almost hundreds of wrong spelling--some of them are exposed to user.
[ https://issues.apache.org/jira/browse/HADOOP-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiXin Ge updated HADOOP-14588: -- Affects Version/s: (was: 3.0.0-alpha4) 3.1.0 > Fix almost hundreds of wrong spelling--some of them are exposed to user. > > > Key: HADOOP-14588 > URL: https://issues.apache.org/jira/browse/HADOOP-14588 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LiXin Ge >Assignee: LiXin Ge >Priority: Major > Attachments: HADOOP-14588.001.patch, HADOOP-14588.002.patch > > > There are almost hundreds of wrong spelling, some wrong spelling in document > file may expose to user. Fix them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14588) Fix almost hundreds of wrong spelling--some of them are exposed to user.
[ https://issues.apache.org/jira/browse/HADOOP-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiXin Ge updated HADOOP-14588: -- Target Version/s: (was: 3.1.0) > Fix almost hundreds of wrong spelling--some of them are exposed to user. > > > Key: HADOOP-14588 > URL: https://issues.apache.org/jira/browse/HADOOP-14588 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LiXin Ge >Assignee: LiXin Ge >Priority: Major > Attachments: HADOOP-14588.001.patch, HADOOP-14588.002.patch > > > There are almost hundreds of wrong spelling, some wrong spelling in document > file may expose to user. Fix them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14588) Fix almost hundreds of wrong spelling--some of them are exposed to user.
[ https://issues.apache.org/jira/browse/HADOOP-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiXin Ge updated HADOOP-14588: -- Status: Open (was: Patch Available) > Fix almost hundreds of wrong spelling--some of them are exposed to user. > > > Key: HADOOP-14588 > URL: https://issues.apache.org/jira/browse/HADOOP-14588 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: LiXin Ge >Assignee: LiXin Ge >Priority: Major > Attachments: HADOOP-14588.001.patch, HADOOP-14588.002.patch > > > There are almost hundreds of wrong spelling, some wrong spelling in document > file may expose to user. Fix them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15222) Refine proxy user authorization to support multiple ACL list
[ https://issues.apache.org/jira/browse/HADOOP-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379563#comment-16379563 ] Eric Yang edited comment on HADOOP-15222 at 2/28/18 1:12 AM: - Today, hadoop offers two roles, cluster admin, and normal users. New system monitor role might be required for separation of duty for service hosting companies. The following table shows a rough sketch of roles required to map to Hadoop web applications: HDFS | /logs | cluster admin | | /jmx | system monitor | | /conf | cluster admin | | /stacks | system monitor | YARN | /logs | cluster admin | | /jmx | system monitor | | /conf | cluster admin | This separation will prevent leaks of customer information. was (Author: eyang): Today, hadoop offers two roles, cluster admin, and normal users. New system admin roles might be required for separation of duty for service hosting companies. The following table shows a rough sketch of roles required to map to Hadoop web applications: HDFS | /logs | cluster admin | | /jmx | system monitor | | /conf | cluster admin | | /stacks | system monitor | YARN | /logs | cluster admin | | /jmx | system monitor | | /conf | cluster admin | This separation will prevent leaks of customer information. > Refine proxy user authorization to support multiple ACL list > > > Key: HADOOP-15222 > URL: https://issues.apache.org/jira/browse/HADOOP-15222 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0 >Reporter: Eric Yang >Priority: Major > > This Jira is responding to follow up work for HADOOP-14077. The original > goal of HADOOP-14077 is to have ability to support multiple ACL lists. The > original problem is a separation of duty use case where the Hadoop cluster > hosting company monitors Hadoop cluster through jmx. Application logs and > hdfs contents should not be visible to hosting company system administrators. > When checking for proxy user authorization in AuthenticationFilter to ensure > there is a way to authorize normal users and admin users using separate proxy > users ACL lists. This was suggested in HADOOP-14060 to configure > AuthenticationFilterWithProxyUser this way: > AuthenticationFilterWithProxyUser->StaticUserWebFilter->AuthenticationFIlterWithProxyUser > This enables the second AuthenticationFilterWithProxyUser validates both > credentials claim by proxy user, and end user. > However, there is a side effect that unauthorized users are not properly > rejected with 403 FORBIDDEN message if there is no other web filter > configured to handle the required authorization work. > This JIRA is intend to discuss the work of HADOOP-14077 by either combine > StaticUserWebFilter + second AuthenticationFilterWithProxyUser into a > AuthorizationFilterWithProxyUser as a final filter to evict unauthorized > user, or revert both HADOOP-14077 and HADOOP-13119 to eliminate the false > positive in user authorization and impersonation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15222) Refine proxy user authorization to support multiple ACL list
[ https://issues.apache.org/jira/browse/HADOOP-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379563#comment-16379563 ] Eric Yang commented on HADOOP-15222: Today, hadoop offers two roles, cluster admin, and normal users. New system admin roles might be required for separation of duty for service hosting companies. The following table shows a rough sketch of roles required to map to Hadoop web applications: HDFS | /logs | cluster admin | | /jmx | system monitor | | /conf | cluster admin | | /stacks | system monitor | YARN | /logs | cluster admin | | /jmx | system monitor | | /conf | cluster admin | This separation will prevent leaks of customer information. > Refine proxy user authorization to support multiple ACL list > > > Key: HADOOP-15222 > URL: https://issues.apache.org/jira/browse/HADOOP-15222 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0 >Reporter: Eric Yang >Priority: Major > > This Jira is responding to follow up work for HADOOP-14077. The original > goal of HADOOP-14077 is to have ability to support multiple ACL lists. The > original problem is a separation of duty use case where the Hadoop cluster > hosting company monitors Hadoop cluster through jmx. Application logs and > hdfs contents should not be visible to hosting company system administrators. > When checking for proxy user authorization in AuthenticationFilter to ensure > there is a way to authorize normal users and admin users using separate proxy > users ACL lists. This was suggested in HADOOP-14060 to configure > AuthenticationFilterWithProxyUser this way: > AuthenticationFilterWithProxyUser->StaticUserWebFilter->AuthenticationFIlterWithProxyUser > This enables the second AuthenticationFilterWithProxyUser validates both > credentials claim by proxy user, and end user. > However, there is a side effect that unauthorized users are not properly > rejected with 403 FORBIDDEN message if there is no other web filter > configured to handle the required authorization work. > This JIRA is intend to discuss the work of HADOOP-14077 by either combine > StaticUserWebFilter + second AuthenticationFilterWithProxyUser into a > AuthorizationFilterWithProxyUser as a final filter to evict unauthorized > user, or revert both HADOOP-14077 and HADOOP-13119 to eliminate the false > positive in user authorization and impersonation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15270) yarn rmadmin -getGroups returns group from which the user has been removed
Sumana Sathish created HADOOP-15270:
---
Summary: yarn rmadmin -getGroups returns group from which the user
has been removed
Key: HADOOP-15270
URL: https://issues.apache.org/jira/browse/HADOOP-15270
Project: Hadoop Common
Issue Type: Bug
Reporter: Sumana Sathish
{code:title= adding group hrt_yarn_rmadmin_test}
sudo su - -c "groupadd hrt_yarn_rmadmin_test" root
{code}
{Code:title=adding user hrt_yarn_rmadmin_test to group hrt_yarn_rmadmin_test}
sudo su - -c "useradd hrt_yarn_rmadmin_test -g hrt_yarn_rmadmin_test" root
{Code}
{Code:title= adding group hrt_yarn_rmadmin_test_group2 }
sudo su - -c "groupadd hrt_yarn_rmadmin_test_group2" root
{Code}
{Code:title=adding user hrt_yarn_rmadmin_test to group
hrt_yarn_rmadmin_test_group2}
sudo su - -c "usermod -a -G hrt_yarn_rmadmin_test_group2 hrt_yarn_rmadmin_test"
root
{Code}
Refresh and getGroups
{code}
yarn rmadmin -refreshUserToGroupsMappings
/usr/hdp/current/hadoop-yarn-client/bin/yarn rmadmin -getGroups
hrt_yarn_rmadmin_test
hrt_yarn_rmadmin_test : hrt_yarn_rmadmin_test hrt_yarn_rmadmin_test_group2
{code}
Delete group hrt_yarn_rmadmin_test_group2 from user hrt_yarn_rmadmin_test and
refresh and do getGroups.
We can still see group hrt_yarn_rmadmin_test_group2
{code}
sudo su - -c "gpasswd -d hrt_yarn_rmadmin_test hrt_yarn_rmadmin_test_group2"
root
{code}
Removing user hrt_yarn_rmadmin_test from group hrt_yarn_rmadmin_test_group2
{code}
bash-4.2$ /usr/hdp/current/hadoop-yarn-client/bin/yarn rmadmin
-refreshUserToGroupsMappings
/usr/hdp/current/hadoop-yarn-client/bin/yarn rmadmin -getGroups
hrt_yarn_rmadmin_test
hrt_yarn_rmadmin_test : hrt_yarn_rmadmin_test hrt_yarn_rmadmin_test_group2
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15178) Generalize NetUtils#wrapException to handle other subclasses with String Constructor
[ https://issues.apache.org/jira/browse/HADOOP-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379443#comment-16379443 ] Ajay Kumar commented on HADOOP-15178: - [~arpitagarwal] thanks for review and commit, [~ste...@apache.org],[~bharatviswa] thanks for review. > Generalize NetUtils#wrapException to handle other subclasses with String > Constructor > > > Key: HADOOP-15178 > URL: https://issues.apache.org/jira/browse/HADOOP-15178 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-15178.001.patch, HADOOP-15178.002.patch, > HADOOP-15178.003.patch > > > NetUtils#wrapException returns an IOException if exception passed to it is > not of type > SocketException,EOFException,NoRouteToHostException,SocketTimeoutException,UnknownHostException,ConnectException,BindException. > By default, it should always return instance (subclass of IOException) of > same type unless a String constructor is not available. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379440#comment-16379440 ] Ajay Kumar commented on HADOOP-15007: - [~leftnoteasy], [~anu] Added patch for branch 3.1 > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Fix For: 3.2.0 > > Attachments: HADOOP-15007-branch-3.1.000.patch, > HADOOP-15007.000.patch, HADOOP-15007.001.patch, HADOOP-15007.002.patch, > HADOOP-15007.003.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14959) DelegationTokenAuthenticator.authenticate() to wrap network exceptions
[
https://issues.apache.org/jira/browse/HADOOP-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379442#comment-16379442
]
Ajay Kumar commented on HADOOP-14959:
-
[~arpitagarwal] thanks for review and commit,
[~ste...@apache.org],[~bharatviswa] thanks for review.
> DelegationTokenAuthenticator.authenticate() to wrap network exceptions
> --
>
> Key: HADOOP-14959
> URL: https://issues.apache.org/jira/browse/HADOOP-14959
> Project: Hadoop Common
> Issue Type: Improvement
> Components: net, security
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Ajay Kumar
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-14959.001.patch, HADOOP-14959.002.patch
>
>
> network errors raised in {{DelegationTokenAuthenticator.authenticate()}}
> aren't being wrapped, so only return the usual limited-value java.net error
> text. using {{NetUtils.wrapException()}} can address that
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HADOOP-15007: Attachment: HADOOP-15007-branch-3.1.000.patch > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Fix For: 3.2.0 > > Attachments: HADOOP-15007-branch-3.1.000.patch, > HADOOP-15007.000.patch, HADOOP-15007.001.patch, HADOOP-15007.002.patch, > HADOOP-15007.003.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15007) Stabilize and document Configuration element
[
https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379387#comment-16379387
]
Wangda Tan commented on HADOOP-15007:
-
[~anu], apologize for my late responses, was traveling in the last a couple of
days and recovering from jet lag
I just found the patch removed class marked {{@Evolving}} which means we *must*
include this patch in 3.1.0 as well (see
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html).
I just reopened this JIRA and set target version to 3.1.0.
To me I'd like to include this improvement in 3.1.0 since it improves doc for
the new added feature, please backport to 3.1.0 if you think the change is safe.
> Stabilize and document Configuration element
> --
>
> Key: HADOOP-15007
> URL: https://issues.apache.org/jira/browse/HADOOP-15007
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Ajay Kumar
>Priority: Blocker
> Fix For: 3.2.0
>
> Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch,
> HADOOP-15007.002.patch, HADOOP-15007.003.patch
>
>
> HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a
> value.
> We need to make sure that this feature is backwards compatible & usable in
> production. That's docs, testing, marshalling etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan reopened HADOOP-15007: - > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Fix For: 3.2.0 > > Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch, > HADOOP-15007.002.patch, HADOOP-15007.003.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated HADOOP-15007: Target Version/s: 3.1.0 (was: 3.2.0) > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Fix For: 3.2.0 > > Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch, > HADOOP-15007.002.patch, HADOOP-15007.003.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15264) AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
[
https://issues.apache.org/jira/browse/HADOOP-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated HADOOP-15264:
Target Version/s: 3.1.0
> AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
> -
>
> Key: HADOOP-15264
> URL: https://issues.apache.org/jira/browse/HADOOP-15264
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15264-001.patch
>
>
> The latest versions of the AWS Shaded SDK are declaring a dependency on netty
> 4.1.17
> {code}
> [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.2.0-SNAPSHOT:compile
> [INFO] | \- com.amazonaws:aws-java-sdk-bundle:jar:1.11.271:compile
> [INFO] | +- io.netty:netty-codec-http:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-codec:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-handler:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-buffer:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-common:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-transport:jar:4.1.17.Final:compile
> [INFO] | \- io.netty:netty-resolver:jar:4.1.17.Final:compile
> {code}
> We either exclude these or roll back HADOOP-15040.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15269) S3 returning 400 on the directory /test/ GET of getFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379327#comment-16379327
]
Aaron Fabbri commented on HADOOP-15269:
---
Do you, or have you ever had, object versioning enabled on this bucket?
I had some specific paths that were throwing errors in the past and it turns
out I had so many object versions accumulated under certain /test paths that s3
was choking on it.
> S3 returning 400 on the directory /test/ GET of getFileStatus
> -
>
> Key: HADOOP-15269
> URL: https://issues.apache.org/jira/browse/HADOOP-15269
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0, 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
>
> Since Monday Feb 26, I'm getting intermittent failures of getFileStatus on a
> directory
> # file path: {{/test}} is returning 404, as expected
> # directory path {{//test/}} is returning 400, so failing the entire operation
> S3 Ireland.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15264) AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
[
https://issues.apache.org/jira/browse/HADOOP-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379319#comment-16379319
]
Aaron Fabbri commented on HADOOP-15264:
---
+1 LGTM. No testing but I pulled trunk of the aws-java-sdk source and searched
for netty. Looks like it is only used for {{kenesisvideo}}:
{noformat}
$ find . -name \*\.java -exec grep -Hni netty "{}" \; | grep -v kinesisvideo |
wc -l
0
{noformat}
> AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
> -
>
> Key: HADOOP-15264
> URL: https://issues.apache.org/jira/browse/HADOOP-15264
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15264-001.patch
>
>
> The latest versions of the AWS Shaded SDK are declaring a dependency on netty
> 4.1.17
> {code}
> [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.2.0-SNAPSHOT:compile
> [INFO] | \- com.amazonaws:aws-java-sdk-bundle:jar:1.11.271:compile
> [INFO] | +- io.netty:netty-codec-http:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-codec:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-handler:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-buffer:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-common:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-transport:jar:4.1.17.Final:compile
> [INFO] | \- io.netty:netty-resolver:jar:4.1.17.Final:compile
> {code}
> We either exclude these or roll back HADOOP-15040.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15264) AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
[
https://issues.apache.org/jira/browse/HADOOP-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379304#comment-16379304
]
Wangda Tan commented on HADOOP-15264:
-
Thanks [~ste...@apache.org], the patch LGTM. Do you want to make any more
changes / verification within this JIRA's scope?
> AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
> -
>
> Key: HADOOP-15264
> URL: https://issues.apache.org/jira/browse/HADOOP-15264
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15264-001.patch
>
>
> The latest versions of the AWS Shaded SDK are declaring a dependency on netty
> 4.1.17
> {code}
> [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.2.0-SNAPSHOT:compile
> [INFO] | \- com.amazonaws:aws-java-sdk-bundle:jar:1.11.271:compile
> [INFO] | +- io.netty:netty-codec-http:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-codec:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-handler:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-buffer:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-common:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-transport:jar:4.1.17.Final:compile
> [INFO] | \- io.netty:netty-resolver:jar:4.1.17.Final:compile
> {code}
> We either exclude these or roll back HADOOP-15040.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15211) Distcp update not preserving root directory permissions
[ https://issues.apache.org/jira/browse/HADOOP-15211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379193#comment-16379193 ] PRASHANT GOLASH commented on HADOOP-15211: -- In case when child directories are *not* present, *distcp copies the permissions* In case when child directories are present, *distcp preserves the permissions* In case when parent directory is *not* present, *distcp copies the permissions* In case when parent directory is present, *distcp does not preserves the permissions* I think ideally the behavior of child and parent should be same as I am passing "pugpb" options > Distcp update not preserving root directory permissions > --- > > Key: HADOOP-15211 > URL: https://issues.apache.org/jira/browse/HADOOP-15211 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 2.6.0 >Reporter: PRASHANT GOLASH >Priority: Minor > > "hadoop distcp -pugpb -update " does not preserve permission for > directory. Although, it preserves permissions for child directories > (/child1 etc.) > Using hadoop-distcp', version:'2.6.0-cdh5.7.2 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15137) ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster
[
https://issues.apache.org/jira/browse/HADOOP-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379153#comment-16379153
]
Bharat Viswanadham edited comment on HADOOP-15137 at 2/27/18 7:25 PM:
--
[~leftnoteasy]
I have verified by looking into the content of jar, now yarn-server-api is
included in the mini cluster.
I don't think any risk in including, as it does not break anything.
was (Author: bharatviswa):
[~leftnoteasy]
I have verified by looking into the content of jar, now yarn-server-api is
included in the mini cluster.
I don't think any risk in including, as it does not break anything.
> ClassNotFoundException:
> org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using
> hadoop-client-minicluster
> --
>
> Key: HADOOP-15137
> URL: https://issues.apache.org/jira/browse/HADOOP-15137
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jeff Zhang
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HADOOP-15137.01.patch, HADOOP-15137.02.patch,
> YARN-7673.00.patch
>
>
> I'd like to use hadoop-client-minicluster for hadoop downstream project, but
> I encounter the following exception when starting hadoop minicluster. And I
> check the hadoop-client-minicluster, it indeed does not have this class. Is
> this something that is missing when packaging the published jar ?
> {code}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/api/DistributedSchedulingAMProtocol
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.createResourceManager(MiniYARNCluster.java:851)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:285)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15137) ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster
[
https://issues.apache.org/jira/browse/HADOOP-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379153#comment-16379153
]
Bharat Viswanadham commented on HADOOP-15137:
-
[~leftnoteasy]
I have verified by looking into the content of jar, now yarn-server-api is
included in the mini cluster.
I don't think any risk in including, as it does not break anything.
> ClassNotFoundException:
> org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using
> hadoop-client-minicluster
> --
>
> Key: HADOOP-15137
> URL: https://issues.apache.org/jira/browse/HADOOP-15137
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jeff Zhang
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HADOOP-15137.01.patch, HADOOP-15137.02.patch,
> YARN-7673.00.patch
>
>
> I'd like to use hadoop-client-minicluster for hadoop downstream project, but
> I encounter the following exception when starting hadoop minicluster. And I
> check the hadoop-client-minicluster, it indeed does not have this class. Is
> this something that is missing when packaging the published jar ?
> {code}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/api/DistributedSchedulingAMProtocol
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.createResourceManager(MiniYARNCluster.java:851)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:285)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14295) Authentication proxy filter may fail authorization because of getRemoteAddr
[
https://issues.apache.org/jira/browse/HADOOP-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated HADOOP-14295:
Target Version/s: 3.2.0 (was: 3.1.0)
> Authentication proxy filter may fail authorization because of getRemoteAddr
> ---
>
> Key: HADOOP-14295
> URL: https://issues.apache.org/jira/browse/HADOOP-14295
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.7.4, 3.0.0-alpha2, 2.8.1
>Reporter: Jeffrey E Rodriguez
>Assignee: Jeffrey E Rodriguez
>Priority: Critical
> Attachments: HADOOP-14295.002.patch, HADOOP-14295.003.patch,
> HADOOP-14295.004.patch, hadoop-14295.001.patch
>
>
> When we turn on Hadoop UI Kerberos and try to access Datanode /logs the proxy
> (Knox) would get an Authorization failure and it hosts would should as
> 127.0.0.1 even though Knox wasn't in local host to Datanode, error message:
> {quote}
> "2017-04-08 07:01:23,029 ERROR security.AuthenticationWithProxyUserFilter
> (AuthenticationWithProxyUserFilter.java:getRemoteUser(94)) - Unable to verify
> proxy user: Unauthorized connection for super-user: knox from IP 127.0.0.1"
> {quote}
> We were able to figure out that Datanode have Jetty listening on localhost
> and that Netty is used to server request to DataNode, this was a measure to
> improve performance because of Netty Async NIO design.
> I propose to add a check for x-forwarded-for header since proxys usually
> inject that header before we do a getRemoteAddr
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15137) ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster
[
https://issues.apache.org/jira/browse/HADOOP-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379141#comment-16379141
]
Wangda Tan commented on HADOOP-15137:
-
Thanks [~rohithsharma].
[~bharatviswa], have you validated the patch? Does minicluster works properly
after the change? Given the patch fixed critical issues, I'd like to include
this in 3.1.0 release. Is there any risk by doing that?
> ClassNotFoundException:
> org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using
> hadoop-client-minicluster
> --
>
> Key: HADOOP-15137
> URL: https://issues.apache.org/jira/browse/HADOOP-15137
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jeff Zhang
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HADOOP-15137.01.patch, HADOOP-15137.02.patch,
> YARN-7673.00.patch
>
>
> I'd like to use hadoop-client-minicluster for hadoop downstream project, but
> I encounter the following exception when starting hadoop minicluster. And I
> check the hadoop-client-minicluster, it indeed does not have this class. Is
> this something that is missing when packaging the published jar ?
> {code}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/api/DistributedSchedulingAMProtocol
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.createResourceManager(MiniYARNCluster.java:851)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:285)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14959) DelegationTokenAuthenticator.authenticate() to wrap network exceptions
[
https://issues.apache.org/jira/browse/HADOOP-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379104#comment-16379104
]
Hudson commented on HADOOP-14959:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13727 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13727/])
HADOOP-14959. DelegationTokenAuthenticator.authenticate() to wrap (arp: rev
137f0d324ad6de40fdf34d50a44c79402043a509)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java
> DelegationTokenAuthenticator.authenticate() to wrap network exceptions
> --
>
> Key: HADOOP-14959
> URL: https://issues.apache.org/jira/browse/HADOOP-14959
> Project: Hadoop Common
> Issue Type: Improvement
> Components: net, security
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Ajay Kumar
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-14959.001.patch, HADOOP-14959.002.patch
>
>
> network errors raised in {{DelegationTokenAuthenticator.authenticate()}}
> aren't being wrapped, so only return the usual limited-value java.net error
> text. using {{NetUtils.wrapException()}} can address that
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15209) DistCp to eliminate needless deletion of files under already-deleted directories
[ https://issues.apache.org/jira/browse/HADOOP-15209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379095#comment-16379095 ] Steve Loughran commented on HADOOP-15209: - FWIW, I just timed a (remote) delete of 8000 files from a to-be-deleted bucket, using fs -rm -r. 2h even with bulk list & delete. Wonder if we should do some more shuffling there > DistCp to eliminate needless deletion of files under already-deleted > directories > > > Key: HADOOP-15209 > URL: https://issues.apache.org/jira/browse/HADOOP-15209 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/distcp >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15209-001.patch, HADOOP-15209-002.patch, > HADOOP-15209-003.patch > > > DistCP issues a delete(file) request even if is underneath an already deleted > directory. This generates needless load on filesystems/object stores, and, if > the store throttles delete, can dramatically slow down the delete operation. > If the distcp delete operation can build a history of deleted directories, > then it will know when it does not need to issue those deletes. > Care is needed here to make sure that whatever structure is created does not > overload the heap of the process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15252) Checkstyle version is not compatible with IDEA's checkstyle plugin
[
https://issues.apache.org/jira/browse/HADOOP-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378995#comment-16378995
]
genericqa commented on HADOOP-15252:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 12m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
62m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 12m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 52s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}197m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage |
| | hadoop.crypto.key.kms.server.TestKMS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15252 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912274/HADOOP-15252.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 054ce23d5d9a 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1e85a99 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14222/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14222/testReport/ |
| Max. process+thread count | 1362 (vs. ulimit of 1) |
| modules | C: hadoop-build-tools . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14222/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This
[jira] [Assigned] (HADOOP-14454) Use reflection to register commands to the command factory
[ https://issues.apache.org/jira/browse/HADOOP-14454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong reassigned HADOOP-14454: --- Assignee: (was: maobaolong) > Use reflection to register commands to the command factory > -- > > Key: HADOOP-14454 > URL: https://issues.apache.org/jira/browse/HADOOP-14454 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 2.7.1, 3.0.0-alpha2 >Reporter: maobaolong >Priority: Minor > > With this feature, when developer develop a new command, he just care the > command class itself -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15262) AliyunOSS: rename() to move files in a directory in parallel
[
https://issues.apache.org/jira/browse/HADOOP-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378964#comment-16378964
]
genericqa commented on HADOOP-15262:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15262 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912297/HADOOP-15262.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 5bda7102cb49 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 28f644b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14226/testReport/ |
| Max. process+thread count | 358 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14226/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AliyunOSS: rename() to move files in a directory in parallel
>
>
> Key: HADOOP-15262
>
[jira] [Updated] (HADOOP-14959) DelegationTokenAuthenticator.authenticate() to wrap network exceptions
[
https://issues.apache.org/jira/browse/HADOOP-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-14959:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
I've committed this. Thanks [~ajayydv], and thanks Steve and Bharat for the
reviews.
> DelegationTokenAuthenticator.authenticate() to wrap network exceptions
> --
>
> Key: HADOOP-14959
> URL: https://issues.apache.org/jira/browse/HADOOP-14959
> Project: Hadoop Common
> Issue Type: Improvement
> Components: net, security
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Ajay Kumar
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-14959.001.patch, HADOOP-14959.002.patch
>
>
> network errors raised in {{DelegationTokenAuthenticator.authenticate()}}
> aren't being wrapped, so only return the usual limited-value java.net error
> text. using {{NetUtils.wrapException()}} can address that
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15269) S3 returning 400 on the directory /test/ GET of getFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15269:
Affects Version/s: 3.1.0
> S3 returning 400 on the directory /test/ GET of getFileStatus
> -
>
> Key: HADOOP-15269
> URL: https://issues.apache.org/jira/browse/HADOOP-15269
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0, 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
>
> Since Monday Feb 26, I'm getting intermittent failures of getFileStatus on a
> directory
> # file path: {{/test}} is returning 404, as expected
> # directory path {{//test/}} is returning 400, so failing the entire operation
> S3 Ireland.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15269) S3 returning 400 on the directory /test/ GET of getFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378903#comment-16378903
]
Steve Loughran commented on HADOOP-15269:
-
Possibly related. Running cloudstore diags against the path
s3a://hwdev-stevel-ireland-new/example4 works (it lists, mkdirs if needed, adds
children), but against /test it fails in DDB/S3Guard
{code}
ard_metadatastore_throttle_rate99thPercentileFrequency (Hz)=0} }}
2018-02-27 16:37:33,753 [main] INFO diag.StoreDiag
(DurationInfo.java:(50)) - Starting: GetFileStatus
s3a://hwdev-steve-ireland-new/
2018-02-27 16:37:33,753 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - op_get_file_status += 1 ->
1
2018-02-27 16:37:33,753 [main] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerGetFileStatus(2098)) - Getting path status for
s3a://hwdev-steve-ireland-new/ ()
2018-02-27 16:37:33,754 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:get(437)) - Get from table hwdev-steve-ireland-new
in region eu-west-1: s3a://hwdev-steve-ireland-new/
root entry S3AFileStatus{path=s3a://hwdev-steve-ireland-new/; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=UNKNOWN
2018-02-27 16:37:33,760 [main] INFO diag.StoreDiag
(DurationInfo.java:close(83)) - GetFileStatus s3a://hwdev-steve-ireland-new/:
duration 0:00:007
2018-02-27 16:37:33,760 [main] INFO diag.StoreDiag
(DurationInfo.java:(50)) - Starting: Listing
s3a://hwdev-steve-ireland-new/
2018-02-27 16:37:33,761 [main] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerListStatus(1876)) - List status for path:
s3a://hwdev-steve-ireland-new/
2018-02-27 16:37:33,761 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - op_list_status += 1 -> 1
2018-02-27 16:37:33,761 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - op_get_file_status += 1 ->
2
2018-02-27 16:37:33,761 [main] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerGetFileStatus(2098)) - Getting path status for
s3a://hwdev-steve-ireland-new/ ()
2018-02-27 16:37:33,761 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:get(437)) - Get from table hwdev-steve-ireland-new
in region eu-west-1: s3a://hwdev-steve-ireland-new/
2018-02-27 16:37:33,761 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:listChildren(505)) - Listing table
hwdev-steve-ireland-new in region eu-west-1: s3a://hwdev-steve-ireland-new/
2018-02-27 16:37:33,810 [main] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerListStatus(1893)) - listStatus: doing listObjects for
directory
2018-02-27 16:37:33,812 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_list_requests += 1
-> 1
2018-02-27 16:37:33,867 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(452)) -
s3a://hwdev-steve-ireland-new/file.txt: file.txt size=0
2018-02-27 16:37:33,867 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(458)) - Adding:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/file.txt; isDirectory=false;
length=0; replication=1; blocksize=33554432; modification_time=1519744033000;
access_time=0; owner=stevel; group=stevel; permission=rw-rw-rw-;
isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}
isEmptyDirectory=FALSE
2018-02-27 16:37:33,868 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(473)) - Adding directory:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/Users; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE
2018-02-27 16:37:33,868 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(473)) - Adding directory:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/example4; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE
2018-02-27 16:37:33,868 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(473)) - Adding directory:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0001; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE
2018-02-27 16:37:33,868 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(473)) - Adding directory:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0002; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
[jira] [Updated] (HADOOP-15262) AliyunOSS: rename() to move files in a directory in parallel
[ https://issues.apache.org/jira/browse/HADOOP-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15262: - Attachment: HADOOP-15262.003.patch > AliyunOSS: rename() to move files in a directory in parallel > > > Key: HADOOP-15262 > URL: https://issues.apache.org/jira/browse/HADOOP-15262 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.1.0, 2.9.1, 3.0.1 > > Attachments: HADOOP-15262.001.patch, HADOOP-15262.002.patch, > HADOOP-15262.003.patch > > > Currently, rename() operation renames files in series. This will be slow if a > directory contains many files. So we can improve this by rename files in > parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15262) AliyunOSS: rename() to move files in a directory in parallel
[ https://issues.apache.org/jira/browse/HADOOP-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378874#comment-16378874 ] wujinhu commented on HADOOP-15262: -- Attach patch file and add more unit tests. > AliyunOSS: rename() to move files in a directory in parallel > > > Key: HADOOP-15262 > URL: https://issues.apache.org/jira/browse/HADOOP-15262 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.1.0, 2.9.1, 3.0.1 > > Attachments: HADOOP-15262.001.patch, HADOOP-15262.002.patch, > HADOOP-15262.003.patch > > > Currently, rename() operation renames files in series. This will be slow if a > directory contains many files. So we can improve this by rename files in > parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15262) AliyunOSS: rename() to move files in a directory in parallel
[
https://issues.apache.org/jira/browse/HADOOP-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378872#comment-16378872
]
genericqa commented on HADOOP-15262:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15262 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912288/HADOOP-15262.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 33e3f414f147 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1e85a99 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14225/testReport/ |
| Max. process+thread count | 301 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14225/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AliyunOSS: rename() to move files in a directory in parallel
>
>
> Key: HADOOP-15262
>
[jira] [Commented] (HADOOP-15269) S3 returning 400 on the directory /test/ GET of getFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378863#comment-16378863
]
Ewan Higgs commented on HADOOP-15269:
-
For folks reading along, the netty issue is HADOOP-15264.
> S3 returning 400 on the directory /test/ GET of getFileStatus
> -
>
> Key: HADOOP-15269
> URL: https://issues.apache.org/jira/browse/HADOOP-15269
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
>
> Since Monday Feb 26, I'm getting intermittent failures of getFileStatus on a
> directory
> # file path: {{/test}} is returning 404, as expected
> # directory path {{//test/}} is returning 400, so failing the entire operation
> S3 Ireland.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15269) S3 returning 400 on the directory /test/ GET of getFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378857#comment-16378857
]
Steve Loughran commented on HADOOP-15269:
-
+[~fabbri], [~ehiggs]
# not related to me trying to exclude netty; seeing it on unmodified trunk.
Repeatedly
# going to see if happens for me on branch-3.1 (update: yes. Moving)
# I'm going to play with different endpoints here and see if I can make it go
away
# then different SDK versions
> S3 returning 400 on the directory /test/ GET of getFileStatus
> -
>
> Key: HADOOP-15269
> URL: https://issues.apache.org/jira/browse/HADOOP-15269
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
>
> Since Monday Feb 26, I'm getting intermittent failures of getFileStatus on a
> directory
> # file path: {{/test}} is returning 404, as expected
> # directory path {{//test/}} is returning 400, so failing the entire operation
> S3 Ireland.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15107) Prove the correctness of the new committers, or fix where they are not correct
[ https://issues.apache.org/jira/browse/HADOOP-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378856#comment-16378856 ] Steve Loughran commented on HADOOP-15107: - This is ready for review, would like to get it in ASAP, just to wrap things up > Prove the correctness of the new committers, or fix where they are not correct > -- > > Key: HADOOP-15107 > URL: https://issues.apache.org/jira/browse/HADOOP-15107 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15107-001.patch > > > I'm writing about the paper on the committers, one which, being a proper > paper, requires me to show the committers work. > # define the requirements of a "Correct" committed job (this applies to the > FileOutputCommitter too) > # show that the Staging committer meets these requirements (most of this is > implicit in that it uses the V1 FileOutputCommitter to marshall .pendingset > lists from committed tasks to the final destination, where they are read and > committed. > # Show the magic committer also works. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15178) Generalize NetUtils#wrapException to handle other subclasses with String Constructor
[ https://issues.apache.org/jira/browse/HADOOP-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378838#comment-16378838 ] Hudson commented on HADOOP-15178: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13724 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13724/]) HADOOP-15178. Generalize NetUtils#wrapException to handle other (arp: rev 28f644bf25b80a6642ce1f32f1688a8d8f4c7cca) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetUtils.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java > Generalize NetUtils#wrapException to handle other subclasses with String > Constructor > > > Key: HADOOP-15178 > URL: https://issues.apache.org/jira/browse/HADOOP-15178 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-15178.001.patch, HADOOP-15178.002.patch, > HADOOP-15178.003.patch > > > NetUtils#wrapException returns an IOException if exception passed to it is > not of type > SocketException,EOFException,NoRouteToHostException,SocketTimeoutException,UnknownHostException,ConnectException,BindException. > By default, it should always return instance (subclass of IOException) of > same type unless a String constructor is not available. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15269) S3 returning 400 on the directory /test/ GET of getFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reassigned HADOOP-15269:
---
Assignee: Steve Loughran
> S3 returning 400 on the directory /test/ GET of getFileStatus
> -
>
> Key: HADOOP-15269
> URL: https://issues.apache.org/jira/browse/HADOOP-15269
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
>
> Since Monday Feb 26, I'm getting intermittent failures of getFileStatus on a
> directory
> # file path: {{/test}} is returning 404, as expected
> # directory path {{//test/}} is returning 400, so failing the entire operation
> S3 Ireland.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15269) S3 returning 400 on the directory /test/ GET of getFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378836#comment-16378836
]
Steve Loughran commented on HADOOP-15269:
-
Stack.
I am authed, because the initial probe for a bucket works, the probe for the
path /test is being authed & returning the 404. It's the second check for a /
which is failing.
{code}
018-02-27 15:46:37,208 [Thread-0] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:initialize(233)) - Initializing S3AFileSystem for
hwdev-steve-ireland-new
2018-02-27 15:46:37,229 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:propagateBucketOptions(1001)) - Propagating entries under
fs.s3a.bucket.hwdev-steve-ireland-new.
2018-02-27 15:46:37,248 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:propagateBucketOptions(1022)) - Updating fs.s3a.endpoint from
[core-site.xml]
2018-02-27 15:46:37,262 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:propagateBucketOptions(1022)) - Updating
fs.s3a.committer.magic.enabled from [core-site.xml]
2018-02-27 15:46:37,468 [Thread-0] WARN impl.MetricsConfig
(MetricsConfig.java:loadFirst(134)) - Cannot locate configuration: tried
hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
2018-02-27 15:46:37,879 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:createAWSCredentialProviderSet(572)) - For URI
s3a://hwdev-steve-ireland-new//, using credentials AWSCredentialProviderList:
BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider
com.amazonaws.auth.InstanceProfileCredentialsProvider@4d8585f2
2018-02-27 15:46:38,195 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.connection.maximum is 45
2018-02-27 15:46:38,195 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.attempts.maximum is 2
2018-02-27 15:46:38,196 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.connection.establish.timeout
is 1000
2018-02-27 15:46:38,196 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.connection.timeout is 5000
2018-02-27 15:46:38,196 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.socket.send.buffer is 65536
2018-02-27 15:46:38,196 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.socket.recv.buffer is 32678
2018-02-27 15:46:38,199 [Thread-0] DEBUG s3a.S3AFileSystem
(DefaultS3ClientFactory.java:initUserAgent(183)) - Using User-Agent: Hadoop 2.11
2018-02-27 15:46:38,776 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.paging.maximum is 5000
2018-02-27 15:46:38,778 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:longBytesOption(887)) - Value of fs.s3a.block.size is 33554432
2018-02-27 15:46:38,779 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:longBytesOption(887)) - Value of fs.s3a.readahead.range is 524288
2018-02-27 15:46:38,780 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.max.total.tasks is 10
2018-02-27 15:46:38,780 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:longOption(865)) - Value of fs.s3a.threads.keepalivetime is 60
2018-02-27 15:46:38,833 [Thread-0] DEBUG s3a.AWSCredentialProviderList
(AWSCredentialProviderList.java:getCredentials(122)) - Using credentials from
BasicAWSCredentialsProvider
2018-02-27 15:46:38,840 [Thread-0] DEBUG amazonaws.request
(AmazonHttpClient.java:executeOneRequest(1183)) - Sending Request: HEAD
https://hwdev-steve-ireland-new.s3-eu-west-1.amazonaws.com / Headers:
(User-Agent: Hadoop 2.11, aws-sdk-java/1.11.271 Mac_OS_X/10.13.2
Java_HotSpot(TM)_64-Bit_Server_VM/25.121-b13 java/1.8.0_121,
amz-sdk-invocation-id: 5b13f82e-7eef-c4bd-b6b2-5047a82673a2, Content-Type:
application/octet-stream, )
2018-02-27 15:46:39,358 [Thread-0] DEBUG amazonaws.request
(AwsResponseHandlerAdapter.java:handle(87)) - Received successful response:
200, AWS Request ID: null
2018-02-27 15:46:39,358 [Thread-0] DEBUG amazonaws.request
(AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId:
not available
2018-02-27 15:46:39,358 [Thread-0] DEBUG amazonaws.request
(AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID:
not available
2018-02-27 15:46:39,368 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:getEncryptionAlgorithm(1253)) - Data is unencrypted
2018-02-27 15:46:39,369 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:getEncryptionAlgorithm(1256)) - Using SSE-C with empty key
2018-02-27 15:46:39,370 [Thread-0] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:initialize(307)) - Input fadvise policy = normal
2018-02-27 15:46:39,370 [Thread-0] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:initialize(311)) - Filesystem support for magic committers
is enabled
2018-02-27 15:46:39,376 [Thread-0] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.fast.upload.active.blocks is 4
2018-02-27 15:46:39,376 [Thread-0] DEBUG
[jira] [Created] (HADOOP-15269) S3 returning 400 on the directory /test/ GET of getFileStatus
Steve Loughran created HADOOP-15269:
---
Summary: S3 returning 400 on the directory /test/ GET of
getFileStatus
Key: HADOOP-15269
URL: https://issues.apache.org/jira/browse/HADOOP-15269
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.2.0
Reporter: Steve Loughran
Since Monday Feb 26, I'm getting intermittent failures of getFileStatus on a
directory
# file path: {{/test}} is returning 404, as expected
# directory path {{//test/}} is returning 400, so failing the entire operation
S3 Ireland.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15264) AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
[
https://issues.apache.org/jira/browse/HADOOP-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378809#comment-16378809
]
genericqa commented on HADOOP-15264:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
24m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 47s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
9s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15264 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912085/HADOOP-15264-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 10ba4f2dd454 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1e85a99 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14224/testReport/ |
| Max. process+thread count | 408 (vs. ulimit of 1) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14224/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
> -
>
> Key: HADOOP-15264
> URL: https://issues.apache.org/jira/browse/HADOOP-15264
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15264-001.patch
>
[jira] [Updated] (HADOOP-15178) Generalize NetUtils#wrapException to handle other subclasses with String Constructor
[ https://issues.apache.org/jira/browse/HADOOP-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15178: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) +1 I've committed this. Thanks Ajay for the improvement, and thanks Steve and Bharat for the code review. > Generalize NetUtils#wrapException to handle other subclasses with String > Constructor > > > Key: HADOOP-15178 > URL: https://issues.apache.org/jira/browse/HADOOP-15178 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-15178.001.patch, HADOOP-15178.002.patch, > HADOOP-15178.003.patch > > > NetUtils#wrapException returns an IOException if exception passed to it is > not of type > SocketException,EOFException,NoRouteToHostException,SocketTimeoutException,UnknownHostException,ConnectException,BindException. > By default, it should always return instance (subclass of IOException) of > same type unless a String constructor is not available. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15178) Generalize NetUtils#wrapException to handle other subclasses with String Constructor
[ https://issues.apache.org/jira/browse/HADOOP-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15178: --- Summary: Generalize NetUtils#wrapException to handle other subclasses with String Constructor (was: Generalize NetUtils#wrapException to handle other subclasses with String Constructor.) > Generalize NetUtils#wrapException to handle other subclasses with String > Constructor > > > Key: HADOOP-15178 > URL: https://issues.apache.org/jira/browse/HADOOP-15178 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HADOOP-15178.001.patch, HADOOP-15178.002.patch, > HADOOP-15178.003.patch > > > NetUtils#wrapException returns an IOException if exception passed to it is > not of type > SocketException,EOFException,NoRouteToHostException,SocketTimeoutException,UnknownHostException,ConnectException,BindException. > By default, it should always return instance (subclass of IOException) of > same type unless a String constructor is not available. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15262) AliyunOSS: rename() to move files in a directory in parallel
[ https://issues.apache.org/jira/browse/HADOOP-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378777#comment-16378777 ] wujinhu commented on HADOOP-15262: -- Fixed some code style issues. > AliyunOSS: rename() to move files in a directory in parallel > > > Key: HADOOP-15262 > URL: https://issues.apache.org/jira/browse/HADOOP-15262 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.1.0, 2.9.1, 3.0.1 > > Attachments: HADOOP-15262.001.patch, HADOOP-15262.002.patch > > > Currently, rename() operation renames files in series. This will be slow if a > directory contains many files. So we can improve this by rename files in > parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15262) AliyunOSS: rename() to move files in a directory in parallel
[ https://issues.apache.org/jira/browse/HADOOP-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15262: - Attachment: HADOOP-15262.002.patch > AliyunOSS: rename() to move files in a directory in parallel > > > Key: HADOOP-15262 > URL: https://issues.apache.org/jira/browse/HADOOP-15262 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.1.0, 2.9.1, 3.0.1 > > Attachments: HADOOP-15262.001.patch, HADOOP-15262.002.patch > > > Currently, rename() operation renames files in series. This will be slow if a > directory contains many files. So we can improve this by rename files in > parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15262) AliyunOSS: rename() to move files in a directory in parallel
[ https://issues.apache.org/jira/browse/HADOOP-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15262: - Attachment: (was: HADOOP-15262.002.patch) > AliyunOSS: rename() to move files in a directory in parallel > > > Key: HADOOP-15262 > URL: https://issues.apache.org/jira/browse/HADOOP-15262 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.1.0, 2.9.1, 3.0.1 > > Attachments: HADOOP-15262.001.patch > > > Currently, rename() operation renames files in series. This will be slow if a > directory contains many files. So we can improve this by rename files in > parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15262) AliyunOSS: rename() to move files in a directory in parallel
[
https://issues.apache.org/jira/browse/HADOOP-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378764#comment-16378764
]
genericqa commented on HADOOP-15262:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 10s{color} | {color:orange} hadoop-tools/hadoop-aliyun: The patch generated
2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15262 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912277/HADOOP-15262.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 1e3e37e45229 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1e85a99 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14223/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aliyun.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14223/testReport/ |
| Max. process+thread count | 324 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14223/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
[jira] [Commented] (HADOOP-15264) AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
[
https://issues.apache.org/jira/browse/HADOOP-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378749#comment-16378749
]
Steve Loughran commented on HADOOP-15264:
-
Think we need some explicit policy w.r.t JAR updates in our s3a testing docs
Something like
* JAR updates potentially brittle, esp the 1.XX to 1.YY. Even the minor ones
have surprises (+point to links related to abort(), NPEs)
* regression test everything with/without s3guard, scale including assumed roles
* + against > 1 endpoint, especially a v4
* Maybe even try v2 signing
* Seek tests from anyone with internal endpoints. e.g [~ehiggs]
* Don't panic about new deprecation warnings, be selective about whether to
upgrade, as its not always clear the new versions are better in terms of
maintenance
* do a mvn dep of the hadoop-aws and hadoop-cloudstore modules, review, reject
any new additions
* any downstream tests you can pull off too, including my little [cloudstore
diags|https://github.com/steveloughran/cloudstore]
* *review the logs*. Those extra warnings and stack traces didn't cause
problems in the tests, but showed up when we looked at things
> AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
> -
>
> Key: HADOOP-15264
> URL: https://issues.apache.org/jira/browse/HADOOP-15264
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15264-001.patch
>
>
> The latest versions of the AWS Shaded SDK are declaring a dependency on netty
> 4.1.17
> {code}
> [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.2.0-SNAPSHOT:compile
> [INFO] | \- com.amazonaws:aws-java-sdk-bundle:jar:1.11.271:compile
> [INFO] | +- io.netty:netty-codec-http:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-codec:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-handler:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-buffer:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-common:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-transport:jar:4.1.17.Final:compile
> [INFO] | \- io.netty:netty-resolver:jar:4.1.17.Final:compile
> {code}
> We either exclude these or roll back HADOOP-15040.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15264) AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
[
https://issues.apache.org/jira/browse/HADOOP-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reassigned HADOOP-15264:
---
Assignee: Steve Loughran
> AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
> -
>
> Key: HADOOP-15264
> URL: https://issues.apache.org/jira/browse/HADOOP-15264
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15264-001.patch
>
>
> The latest versions of the AWS Shaded SDK are declaring a dependency on netty
> 4.1.17
> {code}
> [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.2.0-SNAPSHOT:compile
> [INFO] | \- com.amazonaws:aws-java-sdk-bundle:jar:1.11.271:compile
> [INFO] | +- io.netty:netty-codec-http:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-codec:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-handler:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-buffer:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-common:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-transport:jar:4.1.17.Final:compile
> [INFO] | \- io.netty:netty-resolver:jar:4.1.17.Final:compile
> {code}
> We either exclude these or roll back HADOOP-15040.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15264) AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
[
https://issues.apache.org/jira/browse/HADOOP-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15264:
Status: Patch Available (was: Open)
> AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
> -
>
> Key: HADOOP-15264
> URL: https://issues.apache.org/jira/browse/HADOOP-15264
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15264-001.patch
>
>
> The latest versions of the AWS Shaded SDK are declaring a dependency on netty
> 4.1.17
> {code}
> [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.2.0-SNAPSHOT:compile
> [INFO] | \- com.amazonaws:aws-java-sdk-bundle:jar:1.11.271:compile
> [INFO] | +- io.netty:netty-codec-http:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-codec:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-handler:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-buffer:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-common:jar:4.1.17.Final:compile
> [INFO] | +- io.netty:netty-transport:jar:4.1.17.Final:compile
> [INFO] | \- io.netty:netty-resolver:jar:4.1.17.Final:compile
> {code}
> We either exclude these or roll back HADOOP-15040.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15262) AliyunOSS: rename() to move files in a directory in parallel
[ https://issues.apache.org/jira/browse/HADOOP-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15262: - Attachment: HADOOP-15262.002.patch > AliyunOSS: rename() to move files in a directory in parallel > > > Key: HADOOP-15262 > URL: https://issues.apache.org/jira/browse/HADOOP-15262 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.1.0, 2.9.1, 3.0.1 > > Attachments: HADOOP-15262.001.patch, HADOOP-15262.002.patch > > > Currently, rename() operation renames files in series. This will be slow if a > directory contains many files. So we can improve this by rename files in > parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15262) AliyunOSS: rename() to move files in a directory in parallel
[
https://issues.apache.org/jira/browse/HADOOP-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378655#comment-16378655
]
genericqa commented on HADOOP-15262:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 10s{color} | {color:orange} hadoop-tools/hadoop-aliyun: The patch generated
59 new + 0 unchanged - 0 fixed = 59 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 33s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15262 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912266/HADOOP-15262.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux f5f754b80aaf 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1e85a99 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14221/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aliyun.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14221/testReport/ |
| Max. process+thread count | 303 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14221/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org
[jira] [Updated] (HADOOP-15252) Checkstyle version is not compatible with IDEA's checkstyle plugin
[
https://issues.apache.org/jira/browse/HADOOP-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Bokor updated HADOOP-15252:
--
Attachment: HADOOP-15252.003.patch
> Checkstyle version is not compatible with IDEA's checkstyle plugin
> --
>
> Key: HADOOP-15252
> URL: https://issues.apache.org/jira/browse/HADOOP-15252
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
> Attachments: HADOOP-15252.001.patch, HADOOP-15252.002.patch,
> HADOOP-15252.003.patch, idea_checkstyle_settings.png
>
>
> After upgrading to the latest IDEA the IDE throws error messages in every few
> minutes like
> {code:java}
> The Checkstyle rules file could not be parsed.
> SuppressionCommentFilter is not allowed as a child in Checker
> The file has been blacklisted for 60s.{code}
> This is caused by some backward incompatible changes in checkstyle source
> code:
> [http://checkstyle.sourceforge.net/releasenotes.html]
> * 8.1: Make SuppressionCommentFilter and SuppressWithNearbyCommentFilter
> children of TreeWalker.
> * 8.2: remove FileContentsHolder module as FileContents object is available
> for filters on TreeWalker in TreeWalkerAudit Event.
> IDEA uses checkstyle 8.8
> We should upgrade our checkstyle version to be compatible with IDEA's
> checkstyle plugin.
> Also it's a good time to upgrade maven-checkstyle-plugin as well to brand
> new 3.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15262) AliyunOSS: rename() to move files in a directory in parallel
[ https://issues.apache.org/jira/browse/HADOOP-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15262: - Status: Patch Available (was: Open) > AliyunOSS: rename() to move files in a directory in parallel > > > Key: HADOOP-15262 > URL: https://issues.apache.org/jira/browse/HADOOP-15262 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.1.0, 2.9.1, 3.0.1 > > Attachments: HADOOP-15262.001.patch > > > Currently, rename() operation renames files in series. This will be slow if a > directory contains many files. So we can improve this by rename files in > parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15262) AliyunOSS: rename() to move files in a directory in parallel
[ https://issues.apache.org/jira/browse/HADOOP-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15262: - Attachment: HADOOP-15262.001.patch > AliyunOSS: rename() to move files in a directory in parallel > > > Key: HADOOP-15262 > URL: https://issues.apache.org/jira/browse/HADOOP-15262 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.1.0, 2.9.1, 3.0.1 > > Attachments: HADOOP-15262.001.patch > > > Currently, rename() operation renames files in series. This will be slow if a > directory contains many files. So we can improve this by rename files in > parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14775) Change junit dependency in parent pom file to junit 5 while maintaining backward compatibility to junit4.
[
https://issues.apache.org/jira/browse/HADOOP-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378475#comment-16378475
]
genericqa commented on HADOOP-14775:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
29m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 41s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14775 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912245/HADOOP-14775.03.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 77b5ad71cf01 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1e85a99 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14219/testReport/ |
| Max. process+thread count | 301 (vs. ulimit of 1) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14219/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Change junit dependency in parent pom file to junit 5 while maintaining
> backward compatibility to junit4.
> --
>
> Key: HADOOP-14775
> URL: https://issues.apache.org/jira/browse/HADOOP-14775
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0-alpha4
>Reporter:
[jira] [Commented] (HADOOP-15261) Upgrade commons-io from 2.4 to 2.5
[ https://issues.apache.org/jira/browse/HADOOP-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378467#comment-16378467 ] Akira Ajisaka commented on HADOOP-15261: LGTM +1, pending Jenkins. > Upgrade commons-io from 2.4 to 2.5 > -- > > Key: HADOOP-15261 > URL: https://issues.apache.org/jira/browse/HADOOP-15261 > Project: Hadoop Common > Issue Type: Bug > Components: minikdc >Affects Versions: 3.0.0-alpha3 >Reporter: PandaMonkey >Priority: Major > Attachments: hadoop.txt > > > Hi, after analyzing hadoop-common-project\hadoop-minikdc\pom.xml, we found > that Hadoop depends on org.apache.kerby:kerb-simplekdc 1.0.1, which > transitivity introduced commons-io:2.5. > At the same time, hadoop directly depends on a older version of > commons-io:2.4. By further look into the source code, these two versions of > commons-io have many different features. The dependency conflict problem > brings high risks of "NotClassDefFoundError:" or "NoSuchMethodError" issues > at runtime. Please notice this problem. Maybe upgrading commons-io from 2.4 > to 2.5 is a good choice. Hope this report can help you. Thanks! > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15261) Upgrade commons-io from 2.4 to 2.5
[ https://issues.apache.org/jira/browse/HADOOP-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-15261: --- Target Version/s: (was: 3.0.0) Status: Patch Available (was: Open) > Upgrade commons-io from 2.4 to 2.5 > -- > > Key: HADOOP-15261 > URL: https://issues.apache.org/jira/browse/HADOOP-15261 > Project: Hadoop Common > Issue Type: Bug > Components: minikdc >Affects Versions: 3.0.0-alpha3 >Reporter: PandaMonkey >Priority: Major > Attachments: hadoop.txt > > > Hi, after analyzing hadoop-common-project\hadoop-minikdc\pom.xml, we found > that Hadoop depends on org.apache.kerby:kerb-simplekdc 1.0.1, which > transitivity introduced commons-io:2.5. > At the same time, hadoop directly depends on a older version of > commons-io:2.4. By further look into the source code, these two versions of > commons-io have many different features. The dependency conflict problem > brings high risks of "NotClassDefFoundError:" or "NoSuchMethodError" issues > at runtime. Please notice this problem. Maybe upgrading commons-io from 2.4 > to 2.5 is a good choice. Hope this report can help you. Thanks! > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12760) sun.misc.Cleaner has moved to a new location in OpenJDK 9
[ https://issues.apache.org/jira/browse/HADOOP-12760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378449#comment-16378449 ] Takanobu Asanuma commented on HADOOP-12760: --- Oh, I understand. Thank you for sharing the information! :) > sun.misc.Cleaner has moved to a new location in OpenJDK 9 > - > > Key: HADOOP-12760 > URL: https://issues.apache.org/jira/browse/HADOOP-12760 > Project: Hadoop Common > Issue Type: Bug >Reporter: Chris Hegarty >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-12760.00.patch, HADOOP-12760.01.patch, > HADOOP-12760.02.patch, HADOOP-12760.03.patch, HADOOP-12760.04.patch, > HADOOP-12760.05.patch, HADOOP-12760.06.patch > > > This is a heads-up: there are upcoming changes in JDK 9 that will require, at > least, a small update to org.apache.hadoop.crypto.CryptoStreamUtils & > org.apache.hadoop.io.nativeio.NativeIO. > OpenJDK issue no. 8148117: "Move sun.misc.Cleaner to jdk.internal.ref" [1], > will move the Cleaner class from sun.misc to jdk.internal.ref. There is > ongoing discussion about the possibility of providing a public supported API, > maybe in the JDK 9 timeframe, for releasing NIO direct buffer native memory, > see the core-libs-dev mail thread [2]. At the very least CryptoStreamUtils & > NativeIO [3] should be updated to have knowledge of the new location of the > JDK Cleaner. > [1] https://bugs.openjdk.java.net/browse/JDK-8148117 > [2] > http://mail.openjdk.java.net/pipermail/core-libs-dev/2016-January/038243.html > [3] https://github.com/apache/hadoop/search?utf8=✓=sun.misc.Cleaner -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12760) sun.misc.Cleaner has moved to a new location in OpenJDK 9
[
https://issues.apache.org/jira/browse/HADOOP-12760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378441#comment-16378441
]
Akira Ajisaka commented on HADOOP-12760:
Thank you for the clarification.
FYI
* Now Apache Hadoop yetus-wrapper fetches Apache Yetus 0.4.0 by default and the
smart-apply-patch is broken. (Fixed by YETUS-598)
* Workaround: Set environment variable {{HADOOP_YETUS_VERSION}} to 0.7.0 and
retry.
> sun.misc.Cleaner has moved to a new location in OpenJDK 9
> -
>
> Key: HADOOP-12760
> URL: https://issues.apache.org/jira/browse/HADOOP-12760
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Chris Hegarty
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-12760.00.patch, HADOOP-12760.01.patch,
> HADOOP-12760.02.patch, HADOOP-12760.03.patch, HADOOP-12760.04.patch,
> HADOOP-12760.05.patch, HADOOP-12760.06.patch
>
>
> This is a heads-up: there are upcoming changes in JDK 9 that will require, at
> least, a small update to org.apache.hadoop.crypto.CryptoStreamUtils &
> org.apache.hadoop.io.nativeio.NativeIO.
> OpenJDK issue no. 8148117: "Move sun.misc.Cleaner to jdk.internal.ref" [1],
> will move the Cleaner class from sun.misc to jdk.internal.ref. There is
> ongoing discussion about the possibility of providing a public supported API,
> maybe in the JDK 9 timeframe, for releasing NIO direct buffer native memory,
> see the core-libs-dev mail thread [2]. At the very least CryptoStreamUtils &
> NativeIO [3] should be updated to have knowledge of the new location of the
> JDK Cleaner.
> [1] https://bugs.openjdk.java.net/browse/JDK-8148117
> [2]
> http://mail.openjdk.java.net/pipermail/core-libs-dev/2016-January/038243.html
> [3] https://github.com/apache/hadoop/search?utf8=✓=sun.misc.Cleaner
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14693) Upgrade JUnit from 4 to 5
[ https://issues.apache.org/jira/browse/HADOOP-14693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378421#comment-16378421 ] Akira Ajisaka commented on HADOOP-14693: Hi [~ste...@apache.org], attached a patch in HADOOP-14775 to run junit4 tests by junit-vintage-engine. You can start playing with it. > Upgrade JUnit from 4 to 5 > - > > Key: HADOOP-14693 > URL: https://issues.apache.org/jira/browse/HADOOP-14693 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Priority: Major > > JUnit 4 does not support Java 9. We need to upgrade this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14775) Change junit dependency in parent pom file to junit 5 while maintaining backward compatibility to junit4.
[ https://issues.apache.org/jira/browse/HADOOP-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-14775: --- Status: Patch Available (was: Open) 03 patch: * rebased * Use JUnit 5.1.0 * Downgrade maven-surefire-plugin to 2.19.1 JUnit 5 does not work with maven-surefire-plugin 2.20, and the JUnit team will not support the version because the version does not work with Java 9. https://github.com/junit-team/junit5/issues/809 > Change junit dependency in parent pom file to junit 5 while maintaining > backward compatibility to junit4. > -- > > Key: HADOOP-14775 > URL: https://issues.apache.org/jira/browse/HADOOP-14775 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha4 >Reporter: Ajay Kumar >Assignee: Akira Ajisaka >Priority: Major > Labels: junit5 > Attachments: HADOOP-14775.01.patch, HADOOP-14775.02.patch, > HADOOP-14775.03.patch > > > Change junit dependency in parent pom file to junit 5 while maintaining > backward compatibility to junit4. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14775) Change junit dependency in parent pom file to junit 5 while maintaining backward compatibility to junit4.
[ https://issues.apache.org/jira/browse/HADOOP-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-14775: --- Attachment: HADOOP-14775.03.patch > Change junit dependency in parent pom file to junit 5 while maintaining > backward compatibility to junit4. > -- > > Key: HADOOP-14775 > URL: https://issues.apache.org/jira/browse/HADOOP-14775 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha4 >Reporter: Ajay Kumar >Priority: Major > Labels: junit5 > Attachments: HADOOP-14775.01.patch, HADOOP-14775.02.patch, > HADOOP-14775.03.patch > > > Change junit dependency in parent pom file to junit 5 while maintaining > backward compatibility to junit4. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14775) Change junit dependency in parent pom file to junit 5 while maintaining backward compatibility to junit4.
[ https://issues.apache.org/jira/browse/HADOOP-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reassigned HADOOP-14775: -- Assignee: Akira Ajisaka > Change junit dependency in parent pom file to junit 5 while maintaining > backward compatibility to junit4. > -- > > Key: HADOOP-14775 > URL: https://issues.apache.org/jira/browse/HADOOP-14775 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha4 >Reporter: Ajay Kumar >Assignee: Akira Ajisaka >Priority: Major > Labels: junit5 > Attachments: HADOOP-14775.01.patch, HADOOP-14775.02.patch, > HADOOP-14775.03.patch > > > Change junit dependency in parent pom file to junit 5 while maintaining > backward compatibility to junit4. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12760) sun.misc.Cleaner has moved to a new location in OpenJDK 9
[ https://issues.apache.org/jira/browse/HADOOP-12760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378403#comment-16378403 ] Takanobu Asanuma commented on HADOOP-12760: --- [~ajisakaa] Sorry, I made a mistake. I used "dev-support/bin/smart-apply-patch HADOOP-12760" but it didn't apply the latest patch. After applying the latest patch manually, the build succeeded. Sorry for confusing you. > sun.misc.Cleaner has moved to a new location in OpenJDK 9 > - > > Key: HADOOP-12760 > URL: https://issues.apache.org/jira/browse/HADOOP-12760 > Project: Hadoop Common > Issue Type: Bug >Reporter: Chris Hegarty >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-12760.00.patch, HADOOP-12760.01.patch, > HADOOP-12760.02.patch, HADOOP-12760.03.patch, HADOOP-12760.04.patch, > HADOOP-12760.05.patch, HADOOP-12760.06.patch > > > This is a heads-up: there are upcoming changes in JDK 9 that will require, at > least, a small update to org.apache.hadoop.crypto.CryptoStreamUtils & > org.apache.hadoop.io.nativeio.NativeIO. > OpenJDK issue no. 8148117: "Move sun.misc.Cleaner to jdk.internal.ref" [1], > will move the Cleaner class from sun.misc to jdk.internal.ref. There is > ongoing discussion about the possibility of providing a public supported API, > maybe in the JDK 9 timeframe, for releasing NIO direct buffer native memory, > see the core-libs-dev mail thread [2]. At the very least CryptoStreamUtils & > NativeIO [3] should be updated to have knowledge of the new location of the > JDK Cleaner. > [1] https://bugs.openjdk.java.net/browse/JDK-8148117 > [2] > http://mail.openjdk.java.net/pipermail/core-libs-dev/2016-January/038243.html > [3] https://github.com/apache/hadoop/search?utf8=✓=sun.misc.Cleaner -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14588) Fix almost hundreds of wrong spelling--some of them are exposed to user.
[ https://issues.apache.org/jira/browse/HADOOP-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiXin Ge updated HADOOP-14588: -- Summary: Fix almost hundreds of wrong spelling--some of them are exposed to user. (was: Fix almost hundreds of wrong spelling --- some of which are exposed to user.) > Fix almost hundreds of wrong spelling--some of them are exposed to user. > > > Key: HADOOP-14588 > URL: https://issues.apache.org/jira/browse/HADOOP-14588 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: LiXin Ge >Assignee: LiXin Ge >Priority: Major > Attachments: HADOOP-14588.001.patch, HADOOP-14588.002.patch > > > There are almost hundreds of wrong spelling, some wrong spelling in document > file may expose to user. Fix them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14959) DelegationTokenAuthenticator.authenticate() to wrap network exceptions
[
https://issues.apache.org/jira/browse/HADOOP-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378227#comment-16378227
]
genericqa commented on HADOOP-14959:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
10s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14959 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12901148/HADOOP-14959.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 61d91814450b 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1e85a99 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14217/testReport/ |
| Max. process+thread count | 1348 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14217/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
>
[jira] [Updated] (HADOOP-14588) Fix almost hundreds of wrong spelling --- some of which are exposed to user.
[ https://issues.apache.org/jira/browse/HADOOP-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiXin Ge updated HADOOP-14588: -- Attachment: HADOOP-14588.002.patch > Fix almost hundreds of wrong spelling --- some of which are exposed to user. > > > Key: HADOOP-14588 > URL: https://issues.apache.org/jira/browse/HADOOP-14588 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: LiXin Ge >Assignee: LiXin Ge >Priority: Major > Attachments: HADOOP-14588.001.patch, HADOOP-14588.002.patch > > > There are almost hundreds of wrong spelling, some wrong spelling in document > file may expose to user. Fix them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14588) Fix almost hundreds of wrong spelling --- some of which are exposed to user.
[ https://issues.apache.org/jira/browse/HADOOP-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiXin Ge updated HADOOP-14588: -- Status: Patch Available (was: Open) > Fix almost hundreds of wrong spelling --- some of which are exposed to user. > > > Key: HADOOP-14588 > URL: https://issues.apache.org/jira/browse/HADOOP-14588 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: LiXin Ge >Assignee: LiXin Ge >Priority: Major > Attachments: HADOOP-14588.001.patch, HADOOP-14588.002.patch > > > There are almost hundreds of wrong spelling, some wrong spelling in document > file may expose to user. Fix them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14588) Fix almost hundreds of wrong spelling --- some of which are exposed to user.
[ https://issues.apache.org/jira/browse/HADOOP-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiXin Ge updated HADOOP-14588: -- Status: Open (was: Patch Available) > Fix almost hundreds of wrong spelling --- some of which are exposed to user. > > > Key: HADOOP-14588 > URL: https://issues.apache.org/jira/browse/HADOOP-14588 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: LiXin Ge >Assignee: LiXin Ge >Priority: Major > Attachments: HADOOP-14588.001.patch > > > There are almost hundreds of wrong spelling, some wrong spelling in document > file may expose to user. Fix them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15110) Gauges are getting logged in exceptions from AutoRenewalThreadForUserCreds
[
https://issues.apache.org/jira/browse/HADOOP-15110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378197#comment-16378197
]
genericqa commented on HADOOP-15110:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
5s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15110 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906564/HADOOP-15110.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 3eaee00fd7c1 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1e85a99 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14216/testReport/ |
| Max. process+thread count | 1718 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14216/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Gauges are getting logged
