[jira] [Updated] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fang zhenyi updated HADOOP-15223:
-
Status: Patch Available (was: Open)
> Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap()
> when available
> -
>
> Key: HADOOP-15223
> URL: https://issues.apache.org/jira/browse/HADOOP-15223
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: fang zhenyi
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15223.001.patch
>
>
> The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often
> causes unchecked assignment and it should be replaced with
> {{Collections.emptySet()}} and {{Collections.emptyMap()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fang zhenyi updated HADOOP-15223:
-
Attachment: HADOOP-15223.001.patch
> Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap()
> when available
> -
>
> Key: HADOOP-15223
> URL: https://issues.apache.org/jira/browse/HADOOP-15223
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: fang zhenyi
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15223.001.patch
>
>
> The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often
> causes unchecked assignment and it should be replaced with
> {{Collections.emptySet()}} and {{Collections.emptyMap()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361891#comment-16361891
]
fang zhenyi commented on HADOOP-15223:
--
Thanks [~ajisakaa] for reporting this.I will attach a patch later.
> Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap()
> when available
> -
>
> Key: HADOOP-15223
> URL: https://issues.apache.org/jira/browse/HADOOP-15223
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: fang zhenyi
>Priority: Minor
> Labels: newbie
>
> The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often
> causes unchecked assignment and it should be replaced with
> {{Collections.emptySet()}} and {{Collections.emptyMap()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fang zhenyi reassigned HADOOP-15223:
Assignee: fang zhenyi
> Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap()
> when available
> -
>
> Key: HADOOP-15223
> URL: https://issues.apache.org/jira/browse/HADOOP-15223
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: fang zhenyi
>Priority: Minor
> Labels: newbie
>
> The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often
> causes unchecked assignment and it should be replaced with
> {{Collections.emptySet()}} and {{Collections.emptyMap()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-15223:
---
Labels: newbie (was: )
> Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap()
> when available
> -
>
> Key: HADOOP-15223
> URL: https://issues.apache.org/jira/browse/HADOOP-15223
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Priority: Minor
> Labels: newbie
>
> The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often
> causes unchecked assignment and it should be replaced with
> {{Collections.emptySet()}} and {{Collections.emptyMap()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361861#comment-16361861
]
Akira Ajisaka edited comment on HADOOP-15223 at 2/13/18 6:08 AM:
-
For example, in the following code, {{Collections.EMPTY_SET}} can be replaced
with {{Collections.emptySet()}} to avoid unchecked assignment warning.
{code:java}
public Set getAllocationTags() {
return Collections.EMPTY_SET;
}{code}
was (Author: ajisakaa):
For example, in the following code, {{Collections.EMPTY_SET}} can be replaced
with {{Collections.emptySet()}} to avoid unchecked assignment.
{code:java}
public Set getAllocationTags() {
return Collections.EMPTY_SET;
}{code}
> Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap()
> when available
> -
>
> Key: HADOOP-15223
> URL: https://issues.apache.org/jira/browse/HADOOP-15223
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Priority: Minor
>
> The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often
> causes unchecked assignment and it should be replaced with
> {{Collections.emptySet()}} and {{Collections.emptyMap()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka moved YARN-7924 to HADOOP-15223:
--
Key: HADOOP-15223 (was: YARN-7924)
Project: Hadoop Common (was: Hadoop YARN)
> Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap()
> when available
> -
>
> Key: HADOOP-15223
> URL: https://issues.apache.org/jira/browse/HADOOP-15223
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Priority: Minor
>
> The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often
> causes unchecked assignment and it should be replaced with
> {{Collections.emptySet()}} and {{Collections.emptyMap()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15195) With SELinux enabled, directories mounted with start-build-env.sh may not be accessible.
[ https://issues.apache.org/jira/browse/HADOOP-15195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361846#comment-16361846 ] Hudson commented on HADOOP-15195: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13650 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13650/]) Revert "HADOOP-15195. With SELinux enabled, directories mounted with (cdouglas: rev 9cc6d1dfb351f505aaa8f9f028068650b3b00d0d) * (delete) hadoop-common-project/hadoop-common/src/test/scripts/start-build-env.bats * (edit) start-build-env.sh HADOOP-15195. With SELinux enabled, directories mounted with (cdouglas: rev 0c5d7d71a80bccd4ad7eab269d0727b999606a7e) * (edit) start-build-env.sh * (add) hadoop-common-project/hadoop-common/src/test/scripts/start-build-env.bats > With SELinux enabled, directories mounted with start-build-env.sh may not be > accessible. > > > Key: HADOOP-15195 > URL: https://issues.apache.org/jira/browse/HADOOP-15195 > Project: Hadoop Common > Issue Type: Improvement > Components: build > Environment: Systems with SELinux enabled, e.g., Red Hat Linux 7, > CentOS 7, Fedora. >Reporter: Grigori Rybkine >Assignee: Grigori Rybkine >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-15195.001.patch, HADOOP-15195.001.patch, > HADOOP-15195.002.patch, HADOOP-15195.003.patch, HADOOP-15195.004.patch, > HADOOP-15195.005.patch > > > On a system with SELinux enabled, e.g., Red Hat Linux 7, CentOS 7, Fedora, > the host directories - with the Hadoop code and the maven .m2 - mounted with > start-build-env.sh may not be accessible to the container. This precludes > Hadoop development on such systems. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15195) With SELinux enabled, directories mounted with start-build-env.sh may not be accessible.
[ https://issues.apache.org/jira/browse/HADOOP-15195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361836#comment-16361836 ] Hudson commented on HADOOP-15195: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13649 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13649/]) HADOOP-15195. With SELinux enabled, directories mounted with (cdouglas: rev 5b88cb339898f82519223bcd07e1caedff02d051) * (add) hadoop-common-project/hadoop-common/src/test/scripts/start-build-env.bats * (edit) start-build-env.sh > With SELinux enabled, directories mounted with start-build-env.sh may not be > accessible. > > > Key: HADOOP-15195 > URL: https://issues.apache.org/jira/browse/HADOOP-15195 > Project: Hadoop Common > Issue Type: Improvement > Components: build > Environment: Systems with SELinux enabled, e.g., Red Hat Linux 7, > CentOS 7, Fedora. >Reporter: Grigori Rybkine >Assignee: Grigori Rybkine >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-15195.001.patch, HADOOP-15195.001.patch, > HADOOP-15195.002.patch, HADOOP-15195.003.patch, HADOOP-15195.004.patch, > HADOOP-15195.005.patch > > > On a system with SELinux enabled, e.g., Red Hat Linux 7, CentOS 7, Fedora, > the host directories - with the Hadoop code and the maven .m2 - mounted with > start-build-env.sh may not be accessible to the container. This precludes > Hadoop development on such systems. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15195) With SELinux enabled, directories mounted with start-build-env.sh may not be accessible.
[ https://issues.apache.org/jira/browse/HADOOP-15195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361831#comment-16361831 ] Chris Douglas commented on HADOOP-15195: Sorry for the revert noise; pushed v003 the first time. > With SELinux enabled, directories mounted with start-build-env.sh may not be > accessible. > > > Key: HADOOP-15195 > URL: https://issues.apache.org/jira/browse/HADOOP-15195 > Project: Hadoop Common > Issue Type: Improvement > Components: build > Environment: Systems with SELinux enabled, e.g., Red Hat Linux 7, > CentOS 7, Fedora. >Reporter: Grigori Rybkine >Assignee: Grigori Rybkine >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-15195.001.patch, HADOOP-15195.001.patch, > HADOOP-15195.002.patch, HADOOP-15195.003.patch, HADOOP-15195.004.patch, > HADOOP-15195.005.patch > > > On a system with SELinux enabled, e.g., Red Hat Linux 7, CentOS 7, Fedora, > the host directories - with the Hadoop code and the maven .m2 - mounted with > start-build-env.sh may not be accessible to the container. This precludes > Hadoop development on such systems. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15195) With SELinux enabled, directories mounted with start-build-env.sh may not be accessible.
[ https://issues.apache.org/jira/browse/HADOOP-15195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-15195: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) +1 Great, thanks for the tests. Committed to trunk and branch-3.1. Thanks [~rybkine] > With SELinux enabled, directories mounted with start-build-env.sh may not be > accessible. > > > Key: HADOOP-15195 > URL: https://issues.apache.org/jira/browse/HADOOP-15195 > Project: Hadoop Common > Issue Type: Improvement > Components: build > Environment: Systems with SELinux enabled, e.g., Red Hat Linux 7, > CentOS 7, Fedora. >Reporter: Grigori Rybkine >Assignee: Grigori Rybkine >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-15195.001.patch, HADOOP-15195.001.patch, > HADOOP-15195.002.patch, HADOOP-15195.003.patch, HADOOP-15195.004.patch, > HADOOP-15195.005.patch > > > On a system with SELinux enabled, e.g., Red Hat Linux 7, CentOS 7, Fedora, > the host directories - with the Hadoop code and the maven .m2 - mounted with > start-build-env.sh may not be accessible to the container. This precludes > Hadoop development on such systems. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15222) Refine proxy user authorization to support multiple ACL list
[ https://issues.apache.org/jira/browse/HADOOP-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361799#comment-16361799 ] Eric Yang commented on HADOOP-15222: [~lmccay] Sorry, until a better proposal is feasible to secure /log and /jmx, there is no good enough reason to justify the revert of HADOOP-13119. [~arpitagarwal]'s report was not valid on HADOOP-13119, and HADOOP-13119 does provide better security for authorized users than anonymous to access /log. I can not agree on the revert on HADOOP-13119 at this time. > Refine proxy user authorization to support multiple ACL list > > > Key: HADOOP-15222 > URL: https://issues.apache.org/jira/browse/HADOOP-15222 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0 >Reporter: Eric Yang >Priority: Major > > This Jira is responding to follow up work for HADOOP-14077. The original > goal of HADOOP-14077 is to have ability to support multiple ACL lists. When > checking for proxy user authorization in AuthenticationFilter to ensure there > is a way to authorize normal users and admin users using separate proxy users > ACL lists. This was suggested in HADOOP-14060 to configure > AuthenticationFilterWithProxyUser this way: > AuthenticationFilterWithProxyUser->StaticUserWebFilter->AuthenticationFIlterWithProxyUser > This enables the second AuthenticationFilterWithProxyUser validates both > credentials claim by proxy user, and end user. > However, there is a side effect that unauthorized users are not properly > rejected with 403 FORBIDDEN message if there is no other web filter > configured to handle the required authorization work. > This JIRA is intend to discuss the work of HADOOP-14077 by either combine > StaticUserWebFilter + second AuthenticationFilterWithProxyUser into a > AuthorizationFilterWithProxyUser as a final filter to evict unauthorized > user, or revert both HADOOP-14077 and HADOOP-13119 to eliminate the false > positive in user authorization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13119) Add ability to secure log servlet using proxy users
[ https://issues.apache.org/jira/browse/HADOOP-13119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361794#comment-16361794 ] Eric Yang commented on HADOOP-13119: [~arpitagarwal] the test case is invalid. Your curl command does not contain --negotiate -u :, and Null user can only happen if HADOOP-14077 is applied. > Add ability to secure log servlet using proxy users > --- > > Key: HADOOP-13119 > URL: https://issues.apache.org/jira/browse/HADOOP-13119 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 2.7.4 >Reporter: Jeffrey E Rodriguez >Assignee: Yuanbo Liu >Priority: Major > Labels: security > Fix For: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2 > > Attachments: HADOOP-13119.001.patch, HADOOP-13119.002.patch, > HADOOP-13119.003.patch, HADOOP-13119.004.patch, HADOOP-13119.005.patch, > HADOOP-13119.005.patch, screenshot-1.png > > > User Hadoop on secure mode. > login as kdc user, kinit. > start firefox and enable Kerberos > access http://localhost:50070/logs/ > Get 403 authorization errors. > only hdfs user could access logs. > Would expect as a user to be able to web interface logs link. > Same results if using curl: > curl -v --negotiate -u tester: http://localhost:50070/logs/ > HTTP/1.1 403 User tester is unauthorized to access this page. > so: > 1. either don't show links if hdfs user is able to access. > 2. provide mechanism to add users to web application realm. > 3. note that we are pass authentication so the issue is authorization to > /logs/ > suspect that /logs/ path is secure in webdescriptor so suspect users by > default don't have access to secure paths. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15222) Refine proxy user authorization to support multiple ACL list
[ https://issues.apache.org/jira/browse/HADOOP-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361756#comment-16361756 ] Larry McCay commented on HADOOP-15222: -- Revert it from all branches and put it back to proper proxyuser rules enforcement. Also, we should add the block of a configurable set of resources so that they can't be accessed via impersonation since impersonation isn't intended for admin users and some resources may be considered sensitive enough to limit to admins. We can then have a discussion on whether we want to extend impersonated to admins or not. I personally don't think we should but perhaps it can be controlled enough with proper config. > Refine proxy user authorization to support multiple ACL list > > > Key: HADOOP-15222 > URL: https://issues.apache.org/jira/browse/HADOOP-15222 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0 >Reporter: Eric Yang >Priority: Major > > This Jira is responding to follow up work for HADOOP-14077. The original > goal of HADOOP-14077 is to have ability to support multiple ACL lists. When > checking for proxy user authorization in AuthenticationFilter to ensure there > is a way to authorize normal users and admin users using separate proxy users > ACL lists. This was suggested in HADOOP-14060 to configure > AuthenticationFilterWithProxyUser this way: > AuthenticationFilterWithProxyUser->StaticUserWebFilter->AuthenticationFIlterWithProxyUser > This enables the second AuthenticationFilterWithProxyUser validates both > credentials claim by proxy user, and end user. > However, there is a side effect that unauthorized users are not properly > rejected with 403 FORBIDDEN message if there is no other web filter > configured to handle the required authorization work. > This JIRA is intend to discuss the work of HADOOP-14077 by either combine > StaticUserWebFilter + second AuthenticationFilterWithProxyUser into a > AuthorizationFilterWithProxyUser as a final filter to evict unauthorized > user, or revert both HADOOP-14077 and HADOOP-13119 to eliminate the false > positive in user authorization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15222) Refine proxy user authorization to support multiple ACL list
[ https://issues.apache.org/jira/browse/HADOOP-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361731#comment-16361731 ] Eric Yang commented on HADOOP-15222: [~lmccay] Thank you for the summary. This is aligned with the original problem statement. Role based ACL in standard J2EE web application would be the right approach to solve the authorization problem. User can describe in web.xml which url resource are allowed by roles. Roles are mapped to groups of users. It would be nice to do the same in Hadoop. Hadoop web applications don't quite follow J2EE design pattern. This made the problem hard to solve for Hadoop. We can start by turning Hadoop jetty Java code back to configuration, and maps to roles. In doing so, we might finish in 2-3 years of hard labour. There might be better ways to resolve this issue that we need to explore. HADOOP-13119 is back ported to Hadoop 2.8.x as a new feature in Hadoop 2.8. Do we revert HADOOP-13119 from 2.8.x or we keep HADOOP-13119 as the temp solution until the new work is completed? > Refine proxy user authorization to support multiple ACL list > > > Key: HADOOP-15222 > URL: https://issues.apache.org/jira/browse/HADOOP-15222 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0 >Reporter: Eric Yang >Priority: Major > > This Jira is responding to follow up work for HADOOP-14077. The original > goal of HADOOP-14077 is to have ability to support multiple ACL lists. When > checking for proxy user authorization in AuthenticationFilter to ensure there > is a way to authorize normal users and admin users using separate proxy users > ACL lists. This was suggested in HADOOP-14060 to configure > AuthenticationFilterWithProxyUser this way: > AuthenticationFilterWithProxyUser->StaticUserWebFilter->AuthenticationFIlterWithProxyUser > This enables the second AuthenticationFilterWithProxyUser validates both > credentials claim by proxy user, and end user. > However, there is a side effect that unauthorized users are not properly > rejected with 403 FORBIDDEN message if there is no other web filter > configured to handle the required authorization work. > This JIRA is intend to discuss the work of HADOOP-14077 by either combine > StaticUserWebFilter + second AuthenticationFilterWithProxyUser into a > AuthorizationFilterWithProxyUser as a final filter to evict unauthorized > user, or revert both HADOOP-14077 and HADOOP-13119 to eliminate the false > positive in user authorization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15206) BZip2 drops and duplicates records when input split size is small
[
https://issues.apache.org/jira/browse/HADOOP-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361615#comment-16361615
]
Jason Lowe commented on HADOOP-15206:
-
Thanks for updating the patch!
bq. In the current implementation, read only "BZ" header when the read mode is
CONTINUOUS. Do you think we should keep this?
Yes, because it's not important to read the header when the codec is in BLOCK
mode. IIUC the main difference between CONTINUOUS and BLOCK mode is that BLOCK
mode will be used when processing splits and CONTINUOUS mode is used when we're
simply trying to decompress the data in one big chunk (i.e.: no splits). BLOCK
mode always will scan for the start of the bz2 block, so it will automatically
skip a bz2 file header while searching for the start of the first bz2 block
from the specified start offset.
Given the splittable codec is always scanning for the block and doesn't really
care what bytes are being skipped, I'm now thinking we can go back to a much
simpler implementation. I think the code can check if we're in BLOCK mode to
know whether we are processing splits or not. If we are in BLOCK mode we avoid
advertising the byte position if start offset is zero just as the previous
patches. In BLOCK mode we should also skip to file offset HEADER_LEN +
SUB_HEADER_LEN + 1 if the start position is >=0 and < HEADER_LEN +
SUB_HEADER_LEN. That will put us one byte past the start of the first bz2
block, and BLOCK mode will automatically scan forward to the next block. This
proposal is very similar to what was implemented in patch 003. I think we just
need to make it only do the position adjustment if we're in BLOCK mode.
> BZip2 drops and duplicates records when input split size is small
> -
>
> Key: HADOOP-15206
> URL: https://issues.apache.org/jira/browse/HADOOP-15206
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.3, 3.0.0
>Reporter: Aki Tanaka
>Priority: Major
> Attachments: HADOOP-15206-test.patch, HADOOP-15206.001.patch,
> HADOOP-15206.002.patch, HADOOP-15206.003.patch, HADOOP-15206.004.patch
>
>
> BZip2 can drop and duplicate record when input split file is small. I
> confirmed that this issue happens when the input split size is between 1byte
> and 4bytes.
> I am seeing the following 2 problem behaviors.
>
> 1. Drop record:
> BZip2 skips the first record in the input file when the input split size is
> small
>
> Set the split size to 3 and tested to load 100 records (0, 1, 2..99)
> {code:java}
> 2018-02-01 10:52:33,502 INFO [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(317)) -
> splits[1]=file:/work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+3
> count=99{code}
> > The input format read only 99 records but not 100 records
>
> 2. Duplicate Record:
> 2 input splits has same BZip2 records when the input split size is small
>
> Set the split size to 1 and tested to load 100 records (0, 1, 2..99)
>
> {code:java}
> 2018-02-01 11:18:49,309 INFO [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(318)) - splits[3]=file
> /work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+1
> count=99
> 2018-02-01 11:18:49,310 WARN [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(308)) - conflict with 1 in split 4
> at position 8
> {code}
>
> I experienced this error when I execute Spark (SparkSQL) job under the
> following conditions:
> * The file size of the input files are small (around 1KB)
> * Hadoop cluster has many slave nodes (able to launch many executor tasks)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15222) Refine proxy user authorization to support multiple ACL list
[ https://issues.apache.org/jira/browse/HADOOP-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361602#comment-16361602 ] Larry McCay commented on HADOOP-15222: -- [~eyang] - IMO, we need to revert both HADOOP-14077 and HADOOP-13119 and then determine whether to address the original issue. Let's please be clear on what that problem is - can you verify whether the following summarizes it properly? # There are deployments that only allow access through a single proxy entry point # Some resources that are accessible through proxies should only be accessible for admins # Proxyuser enforcement is generally used to restrict proxies from impersonating admins and super users for obvious reasons Due to the paradox created by the facts in 2 and 3 above we have the following situation, we need to decide whether we should either: # Disable certain paths for proxy users as they are intended only for direct access by authenticated users and deployments described in #1 above are out of luck # Open the proxyuser enforcement rules to allow admin access for specific paths Personally, I don't believe that the fact that certain resources can't be accessed in deployments that only allow impersonation means that we should redefine the proxyuser enforcement strength. I think that it is valid to consider strengthening the proxyuser enforcement to deny access to specific sensitive resources. Whether or not certain resources are too sensitive for impersonation can be left up to the deployment. > Refine proxy user authorization to support multiple ACL list > > > Key: HADOOP-15222 > URL: https://issues.apache.org/jira/browse/HADOOP-15222 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0 >Reporter: Eric Yang >Priority: Major > > This Jira is responding to follow up work for HADOOP-14077. The original > goal of HADOOP-14077 is to have ability to support multiple ACL lists. When > checking for proxy user authorization in AuthenticationFilter to ensure there > is a way to authorize normal users and admin users using separate proxy users > ACL lists. This was suggested in HADOOP-14060 to configure > AuthenticationFilterWithProxyUser this way: > AuthenticationFilterWithProxyUser->StaticUserWebFilter->AuthenticationFIlterWithProxyUser > This enables the second AuthenticationFilterWithProxyUser validates both > credentials claim by proxy user, and end user. > However, there is a side effect that unauthorized users are not properly > rejected with 403 FORBIDDEN message if there is no other web filter > configured to handle the required authorization work. > This JIRA is intend to discuss the work of HADOOP-14077 by either combine > StaticUserWebFilter + second AuthenticationFilterWithProxyUser into a > AuthorizationFilterWithProxyUser as a final filter to evict unauthorized > user, or revert both HADOOP-14077 and HADOOP-13119 to eliminate the false > positive in user authorization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15178) Generalize NetUtils#wrapException to handle other subclasses with String Constructor.
[
https://issues.apache.org/jira/browse/HADOOP-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361584#comment-16361584
]
genericqa commented on HADOOP-15178:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 49 unchanged - 1 fixed = 49 total (was 50) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 17s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
| | hadoop.ha.TestZKFailoverControllerStress |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15178 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910281/HADOOP-15178.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e021e54adf85 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 87e2570 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14103/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14103/testReport/ |
| Max. process+thread count | 1713 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
|
[jira] [Created] (HADOOP-15222) Refine proxy user authorization to support multiple ACL list
Eric Yang created HADOOP-15222: -- Summary: Refine proxy user authorization to support multiple ACL list Key: HADOOP-15222 URL: https://issues.apache.org/jira/browse/HADOOP-15222 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 3.0.0 Reporter: Eric Yang This Jira is responding to follow up work for HADOOP-14077. The original goal of HADOOP-14077 is to have ability to support multiple ACL lists. When checking for proxy user authorization in AuthenticationFilter to ensure there is a way to authorize normal users and admin users using separate proxy users ACL lists. This was suggested in HADOOP-14060 to configure AuthenticationFilterWithProxyUser this way: AuthenticationFilterWithProxyUser->StaticUserWebFilter->AuthenticationFIlterWithProxyUser This enables the second AuthenticationFilterWithProxyUser validates both credentials claim by proxy user, and end user. However, there is a side effect that unauthorized users are not properly rejected with 403 FORBIDDEN message if there is no other web filter configured to handle the required authorization work. This JIRA is intend to discuss the work of HADOOP-14077 by either combine StaticUserWebFilter + second AuthenticationFilterWithProxyUser into a AuthorizationFilterWithProxyUser as a final filter to evict unauthorized user, or revert both HADOOP-14077 and HADOOP-13119 to eliminate the false positive in user authorization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15129) Datanode caches namenode DNS lookup failure and cannot startup
[
https://issues.apache.org/jira/browse/HADOOP-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361506#comment-16361506
]
Ajay Kumar edited comment on HADOOP-15129 at 2/12/18 9:54 PM:
--
{quote} The local IP address for the connection is indeterminate until the
target hostname is resolved e.g. multi-homed setups. So I am okay with leaving
this as null for now.{quote}
[~arpitagarwal], I see your point. I was thinking that localhost "UNKOWN" may
mislead someone in wrong diagnosis about the problem. Wondering if we can we
improve the messaging. But this can be discussed separately and doesn't need to
be part of this jira.
was (Author: ajayydv):
{quote} The local IP address for the connection is indeterminate until the
target hostname is resolved e.g. multi-homed setups. So I am okay with leaving
this as null for now.{quote}
I see your point. I was thinking that localhost "UNKOWN" may mislead someone in
wrong diagnosis about the problem. Wondering if we can we improve the
messaging. But this can be discussed separately and doesn't need to be part of
this jira.
> Datanode caches namenode DNS lookup failure and cannot startup
> --
>
> Key: HADOOP-15129
> URL: https://issues.apache.org/jira/browse/HADOOP-15129
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.2
> Environment: Google Compute Engine.
> I'm using Java 8, Debian 8, Hadoop 2.8.2.
>Reporter: Karthik Palaniappan
>Assignee: Karthik Palaniappan
>Priority: Minor
> Attachments: HADOOP-15129.001.patch, HADOOP-15129.002.patch
>
>
> On startup, the Datanode creates an InetSocketAddress to register with each
> namenode. Though there are retries on connection failure throughout the
> stack, the same InetSocketAddress is reused.
> InetSocketAddress is an interesting class, because it resolves DNS names to
> IP addresses on construction, and it is never refreshed. Hadoop re-creates an
> InetSocketAddress in some cases just in case the remote IP has changed for a
> particular DNS name: https://issues.apache.org/jira/browse/HADOOP-7472.
> Anyway, on startup, you cna see the Datanode log: "Namenode...remains
> unresolved" -- referring to the fact that DNS lookup failed.
> {code:java}
> 2017-11-02 16:01:55,115 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Refresh request received for nameservices: null
> 2017-11-02 16:01:55,153 WARN org.apache.hadoop.hdfs.DFSUtilClient: Namenode
> for null remains unresolved for ID null. Check your hdfs-site.xml file to
> ensure namenodes are configured properly.
> 2017-11-02 16:01:55,156 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Starting BPOfferServices for nameservices:
> 2017-11-02 16:01:55,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block pool (Datanode Uuid unassigned) service to
> cluster-32f5-m:8020 starting to offer service
> {code}
> The Datanode then proceeds to use this unresolved address, as it may work if
> the DN is configured to use a proxy. Since I'm not using a proxy, it forever
> prints out this message:
> {code:java}
> 2017-12-15 00:13:40,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:45,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:50,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:55,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:14:00,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> {code}
> Unfortunately, the log doesn't contain the exception that triggered it, but
> the culprit is actually in IPC Client:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L444.
> This line was introduced in https://issues.apache.org/jira/browse/HADOOP-487
> to give a clear error message when somebody mispells an address.
> However, the fix in HADOOP-7472 doesn't apply here, because that code happens
> in Client#getConnection after the Connection is constructed.
> My proposed fix (will attach a patch) is to move this exception out of the
> constructor and into a place that will trigger HADOOP-7472's logic to
> re-resolve addresses. If the DNS failure was temporary, this will allow the
> connection to succeed. If not, the connection will fail after ipc client
> retries (default 10 seconds worth of retries).
> I want to fix this in ipc client rather than just in Datanode startup, as
>
[jira] [Commented] (HADOOP-15129) Datanode caches namenode DNS lookup failure and cannot startup
[
https://issues.apache.org/jira/browse/HADOOP-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361506#comment-16361506
]
Ajay Kumar commented on HADOOP-15129:
-
{quote} The local IP address for the connection is indeterminate until the
target hostname is resolved e.g. multi-homed setups. So I am okay with leaving
this as null for now.{quote}
I see your point. I was thinking that localhost "UNKOWN" may mislead someone in
wrong diagnosis about the problem. Wondering if we can we improve the
messaging. But this can be discussed separately and doesn't need to be part of
this jira.
> Datanode caches namenode DNS lookup failure and cannot startup
> --
>
> Key: HADOOP-15129
> URL: https://issues.apache.org/jira/browse/HADOOP-15129
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.2
> Environment: Google Compute Engine.
> I'm using Java 8, Debian 8, Hadoop 2.8.2.
>Reporter: Karthik Palaniappan
>Assignee: Karthik Palaniappan
>Priority: Minor
> Attachments: HADOOP-15129.001.patch, HADOOP-15129.002.patch
>
>
> On startup, the Datanode creates an InetSocketAddress to register with each
> namenode. Though there are retries on connection failure throughout the
> stack, the same InetSocketAddress is reused.
> InetSocketAddress is an interesting class, because it resolves DNS names to
> IP addresses on construction, and it is never refreshed. Hadoop re-creates an
> InetSocketAddress in some cases just in case the remote IP has changed for a
> particular DNS name: https://issues.apache.org/jira/browse/HADOOP-7472.
> Anyway, on startup, you cna see the Datanode log: "Namenode...remains
> unresolved" -- referring to the fact that DNS lookup failed.
> {code:java}
> 2017-11-02 16:01:55,115 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Refresh request received for nameservices: null
> 2017-11-02 16:01:55,153 WARN org.apache.hadoop.hdfs.DFSUtilClient: Namenode
> for null remains unresolved for ID null. Check your hdfs-site.xml file to
> ensure namenodes are configured properly.
> 2017-11-02 16:01:55,156 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Starting BPOfferServices for nameservices:
> 2017-11-02 16:01:55,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block pool (Datanode Uuid unassigned) service to
> cluster-32f5-m:8020 starting to offer service
> {code}
> The Datanode then proceeds to use this unresolved address, as it may work if
> the DN is configured to use a proxy. Since I'm not using a proxy, it forever
> prints out this message:
> {code:java}
> 2017-12-15 00:13:40,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:45,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:50,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:55,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:14:00,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> {code}
> Unfortunately, the log doesn't contain the exception that triggered it, but
> the culprit is actually in IPC Client:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L444.
> This line was introduced in https://issues.apache.org/jira/browse/HADOOP-487
> to give a clear error message when somebody mispells an address.
> However, the fix in HADOOP-7472 doesn't apply here, because that code happens
> in Client#getConnection after the Connection is constructed.
> My proposed fix (will attach a patch) is to move this exception out of the
> constructor and into a place that will trigger HADOOP-7472's logic to
> re-resolve addresses. If the DNS failure was temporary, this will allow the
> connection to succeed. If not, the connection will fail after ipc client
> retries (default 10 seconds worth of retries).
> I want to fix this in ipc client rather than just in Datanode startup, as
> this fixes temporary DNS issues for all of Hadoop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15178) Generalize NetUtils#wrapException to handle other subclasses with String Constructor.
[ https://issues.apache.org/jira/browse/HADOOP-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361494#comment-16361494 ] Ajay Kumar commented on HADOOP-15178: - Rebased for trunk. > Generalize NetUtils#wrapException to handle other subclasses with String > Constructor. > - > > Key: HADOOP-15178 > URL: https://issues.apache.org/jira/browse/HADOOP-15178 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HADOOP-15178.001.patch, HADOOP-15178.002.patch, > HADOOP-15178.003.patch > > > NetUtils#wrapException returns an IOException if exception passed to it is > not of type > SocketException,EOFException,NoRouteToHostException,SocketTimeoutException,UnknownHostException,ConnectException,BindException. > By default, it should always return instance (subclass of IOException) of > same type unless a String constructor is not available. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15178) Generalize NetUtils#wrapException to handle other subclasses with String Constructor.
[ https://issues.apache.org/jira/browse/HADOOP-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HADOOP-15178: Attachment: HADOOP-15178.003.patch > Generalize NetUtils#wrapException to handle other subclasses with String > Constructor. > - > > Key: HADOOP-15178 > URL: https://issues.apache.org/jira/browse/HADOOP-15178 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HADOOP-15178.001.patch, HADOOP-15178.002.patch, > HADOOP-15178.003.patch > > > NetUtils#wrapException returns an IOException if exception passed to it is > not of type > SocketException,EOFException,NoRouteToHostException,SocketTimeoutException,UnknownHostException,ConnectException,BindException. > By default, it should always return instance (subclass of IOException) of > same type unless a String constructor is not available. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15112) create-release didn't sign artifacts
[
https://issues.apache.org/jira/browse/HADOOP-15112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361434#comment-16361434
]
genericqa commented on HADOOP-15112:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
2s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
14s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15112 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910275/HADOOP-15112.01.patch
|
| Optional Tests | asflicense shellcheck shelldocs |
| uname | Linux c4164f565c36 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 87e2570 |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| Max. process+thread count | 301 (vs. ulimit of 5500) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14102/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> create-release didn't sign artifacts
>
>
> Key: HADOOP-15112
> URL: https://issues.apache.org/jira/browse/HADOOP-15112
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Andrew Wang
>Assignee: Lei (Eddy) Xu
>Priority: Major
> Attachments: HADOOP-15112.01.patch
>
>
> While building the 3.0.0 RC1, I had to re-invoke Maven because the
> create-release script didn't deploy signatures to Nexus. Looking at the repo
> (and my artifacts), it seems like "sign" didn't run properly.
> I lost my create-release output, but I noticed that it will log and continue
> rather than abort in some error conditions. This might have caused my lack of
> signatures. IMO it'd be better to explicitly fail in these situations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14961) Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed
[
https://issues.apache.org/jira/browse/HADOOP-14961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge resolved HADOOP-14961.
-
Resolution: Duplicate
Fixed by HADOOP-14816.
> Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed
> --
>
> Key: HADOOP-14961
> URL: https://issues.apache.org/jira/browse/HADOOP-14961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, test
>Affects Versions: 3.1.0
>Reporter: John Zhuge
>Priority: Major
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/13546/console
> {noformat}
> Downloading Oracle Java 8...
> [0m[91m--2017-10-18 18:28:11--
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
>
> [0m[91mResolving download.oracle.com (download.oracle.com)...
> [0m[91m23.59.190.131, 23.59.190.130
> Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80...
> [0m[91mconnected.
> HTTP request sent, awaiting response... [0m[91m302 Moved Temporarily
> Location:
> https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
> [following]
> --2017-10-18 18:28:11--
> https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
>
> [0m[91mResolving edelivery.oracle.com (edelivery.oracle.com)...
> [0m[91m23.39.16.136, 2600:1409:a:39c::2d3e, 2600:1409:a:39e::2d3e
> Connecting to edelivery.oracle.com
> (edelivery.oracle.com)|23.39.16.136|:443... [0m[91mconnected.
> [0m[91mHTTP request sent, awaiting response... [0m[91m302 Moved
> Temporarily
> Location:
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c
> [following]
> --2017-10-18 18:28:11--
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c
>
> Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80...
> [0m[91mconnected.
> HTTP request sent, awaiting response... [0m[91m404 Not Found
> [0m[91m2017-10-18 18:28:12 ERROR 404: Not Found.
> [0m[91mdownload failed
> Oracle JDK 8 is NOT installed.
> {noformat}
> Looks like Oracle JDK 8u144 is no longer available for download using that
> link. 8u151 and 8u152 are available.
> Many of last 10 https://builds.apache.org/job/PreCommit-HADOOP-Build/ jobs
> failed the same way, all on build host H1 and H6.
> [~aw] has a patch available in HADOOP-14816 "Update Dockerfile to use Xenial"
> for a long term fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14961) Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed
[
https://issues.apache.org/jira/browse/HADOOP-14961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge reopened HADOOP-14961:
-
> Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed
> --
>
> Key: HADOOP-14961
> URL: https://issues.apache.org/jira/browse/HADOOP-14961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, test
>Affects Versions: 3.1.0
>Reporter: John Zhuge
>Priority: Major
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/13546/console
> {noformat}
> Downloading Oracle Java 8...
> [0m[91m--2017-10-18 18:28:11--
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
>
> [0m[91mResolving download.oracle.com (download.oracle.com)...
> [0m[91m23.59.190.131, 23.59.190.130
> Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80...
> [0m[91mconnected.
> HTTP request sent, awaiting response... [0m[91m302 Moved Temporarily
> Location:
> https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
> [following]
> --2017-10-18 18:28:11--
> https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
>
> [0m[91mResolving edelivery.oracle.com (edelivery.oracle.com)...
> [0m[91m23.39.16.136, 2600:1409:a:39c::2d3e, 2600:1409:a:39e::2d3e
> Connecting to edelivery.oracle.com
> (edelivery.oracle.com)|23.39.16.136|:443... [0m[91mconnected.
> [0m[91mHTTP request sent, awaiting response... [0m[91m302 Moved
> Temporarily
> Location:
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c
> [following]
> --2017-10-18 18:28:11--
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c
>
> Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80...
> [0m[91mconnected.
> HTTP request sent, awaiting response... [0m[91m404 Not Found
> [0m[91m2017-10-18 18:28:12 ERROR 404: Not Found.
> [0m[91mdownload failed
> Oracle JDK 8 is NOT installed.
> {noformat}
> Looks like Oracle JDK 8u144 is no longer available for download using that
> link. 8u151 and 8u152 are available.
> Many of last 10 https://builds.apache.org/job/PreCommit-HADOOP-Build/ jobs
> failed the same way, all on build host H1 and H6.
> [~aw] has a patch available in HADOOP-14816 "Update Dockerfile to use Xenial"
> for a long term fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15112) create-release didn't sign artifacts
[ https://issues.apache.org/jira/browse/HADOOP-15112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HADOOP-15112: --- Status: Patch Available (was: Open) > create-release didn't sign artifacts > > > Key: HADOOP-15112 > URL: https://issues.apache.org/jira/browse/HADOOP-15112 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Lei (Eddy) Xu >Priority: Major > Attachments: HADOOP-15112.01.patch > > > While building the 3.0.0 RC1, I had to re-invoke Maven because the > create-release script didn't deploy signatures to Nexus. Looking at the repo > (and my artifacts), it seems like "sign" didn't run properly. > I lost my create-release output, but I noticed that it will log and continue > rather than abort in some error conditions. This might have caused my lack of > signatures. IMO it'd be better to explicitly fail in these situations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15112) create-release didn't sign artifacts
[ https://issues.apache.org/jira/browse/HADOOP-15112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HADOOP-15112: --- Attachment: HADOOP-15112.01.patch > create-release didn't sign artifacts > > > Key: HADOOP-15112 > URL: https://issues.apache.org/jira/browse/HADOOP-15112 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Lei (Eddy) Xu >Priority: Major > Attachments: HADOOP-15112.01.patch > > > While building the 3.0.0 RC1, I had to re-invoke Maven because the > create-release script didn't deploy signatures to Nexus. Looking at the repo > (and my artifacts), it seems like "sign" didn't run properly. > I lost my create-release output, but I noticed that it will log and continue > rather than abort in some error conditions. This might have caused my lack of > signatures. IMO it'd be better to explicitly fail in these situations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15129) Datanode caches namenode DNS lookup failure and cannot startup
[
https://issues.apache.org/jira/browse/HADOOP-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361358#comment-16361358
]
Arpit Agarwal commented on HADOOP-15129:
The test case {{testIpcFlakyHostResolution}} also lgtm. The other new test
{{testIpcHostResolutionTimeout}} looks related to this change. Do we need it
[~Karthik Palaniappan]?
bq. can we pass "localhost" for NetUtils.wrapException instead of null. Above
message in logs is little misleading.
Good point. The local IP address for the connection is indeterminate until the
target hostname is resolved e.g. multi-homed setups. So I am okay with leaving
this as null for now. [~ajayydv], let me know if that sounds reasonable.
> Datanode caches namenode DNS lookup failure and cannot startup
> --
>
> Key: HADOOP-15129
> URL: https://issues.apache.org/jira/browse/HADOOP-15129
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.2
> Environment: Google Compute Engine.
> I'm using Java 8, Debian 8, Hadoop 2.8.2.
>Reporter: Karthik Palaniappan
>Assignee: Karthik Palaniappan
>Priority: Minor
> Attachments: HADOOP-15129.001.patch, HADOOP-15129.002.patch
>
>
> On startup, the Datanode creates an InetSocketAddress to register with each
> namenode. Though there are retries on connection failure throughout the
> stack, the same InetSocketAddress is reused.
> InetSocketAddress is an interesting class, because it resolves DNS names to
> IP addresses on construction, and it is never refreshed. Hadoop re-creates an
> InetSocketAddress in some cases just in case the remote IP has changed for a
> particular DNS name: https://issues.apache.org/jira/browse/HADOOP-7472.
> Anyway, on startup, you cna see the Datanode log: "Namenode...remains
> unresolved" -- referring to the fact that DNS lookup failed.
> {code:java}
> 2017-11-02 16:01:55,115 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Refresh request received for nameservices: null
> 2017-11-02 16:01:55,153 WARN org.apache.hadoop.hdfs.DFSUtilClient: Namenode
> for null remains unresolved for ID null. Check your hdfs-site.xml file to
> ensure namenodes are configured properly.
> 2017-11-02 16:01:55,156 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Starting BPOfferServices for nameservices:
> 2017-11-02 16:01:55,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block pool (Datanode Uuid unassigned) service to
> cluster-32f5-m:8020 starting to offer service
> {code}
> The Datanode then proceeds to use this unresolved address, as it may work if
> the DN is configured to use a proxy. Since I'm not using a proxy, it forever
> prints out this message:
> {code:java}
> 2017-12-15 00:13:40,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:45,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:50,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:55,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:14:00,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> {code}
> Unfortunately, the log doesn't contain the exception that triggered it, but
> the culprit is actually in IPC Client:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L444.
> This line was introduced in https://issues.apache.org/jira/browse/HADOOP-487
> to give a clear error message when somebody mispells an address.
> However, the fix in HADOOP-7472 doesn't apply here, because that code happens
> in Client#getConnection after the Connection is constructed.
> My proposed fix (will attach a patch) is to move this exception out of the
> constructor and into a place that will trigger HADOOP-7472's logic to
> re-resolve addresses. If the DNS failure was temporary, this will allow the
> connection to succeed. If not, the connection will fail after ipc client
> retries (default 10 seconds worth of retries).
> I want to fix this in ipc client rather than just in Datanode startup, as
> this fixes temporary DNS issues for all of Hadoop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15129) Datanode caches namenode DNS lookup failure and cannot startup
[
https://issues.apache.org/jira/browse/HADOOP-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361358#comment-16361358
]
Arpit Agarwal edited comment on HADOOP-15129 at 2/12/18 7:53 PM:
-
The test case {{testIpcFlakyHostResolution}} also lgtm. The other new test
{{testIpcHostResolutionTimeout}} looks unrelated to this change. Do we need it
[~Karthik Palaniappan]?
bq. can we pass "localhost" for NetUtils.wrapException instead of null. Above
message in logs is little misleading.
Good point. The local IP address for the connection is indeterminate until the
target hostname is resolved e.g. multi-homed setups. So I am okay with leaving
this as null for now. [~ajayydv], let me know if that sounds reasonable.
was (Author: arpitagarwal):
The test case {{testIpcFlakyHostResolution}} also lgtm. The other new test
{{testIpcHostResolutionTimeout}} looks related to this change. Do we need it
[~Karthik Palaniappan]?
bq. can we pass "localhost" for NetUtils.wrapException instead of null. Above
message in logs is little misleading.
Good point. The local IP address for the connection is indeterminate until the
target hostname is resolved e.g. multi-homed setups. So I am okay with leaving
this as null for now. [~ajayydv], let me know if that sounds reasonable.
> Datanode caches namenode DNS lookup failure and cannot startup
> --
>
> Key: HADOOP-15129
> URL: https://issues.apache.org/jira/browse/HADOOP-15129
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.2
> Environment: Google Compute Engine.
> I'm using Java 8, Debian 8, Hadoop 2.8.2.
>Reporter: Karthik Palaniappan
>Assignee: Karthik Palaniappan
>Priority: Minor
> Attachments: HADOOP-15129.001.patch, HADOOP-15129.002.patch
>
>
> On startup, the Datanode creates an InetSocketAddress to register with each
> namenode. Though there are retries on connection failure throughout the
> stack, the same InetSocketAddress is reused.
> InetSocketAddress is an interesting class, because it resolves DNS names to
> IP addresses on construction, and it is never refreshed. Hadoop re-creates an
> InetSocketAddress in some cases just in case the remote IP has changed for a
> particular DNS name: https://issues.apache.org/jira/browse/HADOOP-7472.
> Anyway, on startup, you cna see the Datanode log: "Namenode...remains
> unresolved" -- referring to the fact that DNS lookup failed.
> {code:java}
> 2017-11-02 16:01:55,115 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Refresh request received for nameservices: null
> 2017-11-02 16:01:55,153 WARN org.apache.hadoop.hdfs.DFSUtilClient: Namenode
> for null remains unresolved for ID null. Check your hdfs-site.xml file to
> ensure namenodes are configured properly.
> 2017-11-02 16:01:55,156 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Starting BPOfferServices for nameservices:
> 2017-11-02 16:01:55,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block pool (Datanode Uuid unassigned) service to
> cluster-32f5-m:8020 starting to offer service
> {code}
> The Datanode then proceeds to use this unresolved address, as it may work if
> the DN is configured to use a proxy. Since I'm not using a proxy, it forever
> prints out this message:
> {code:java}
> 2017-12-15 00:13:40,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:45,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:50,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:55,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:14:00,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> {code}
> Unfortunately, the log doesn't contain the exception that triggered it, but
> the culprit is actually in IPC Client:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L444.
> This line was introduced in https://issues.apache.org/jira/browse/HADOOP-487
> to give a clear error message when somebody mispells an address.
> However, the fix in HADOOP-7472 doesn't apply here, because that code happens
> in Client#getConnection after the Connection is constructed.
> My proposed fix (will attach a patch) is to move this exception out of the
> constructor and into a place that will trigger HADOOP-7472's logic to
> re-resolve addresses. If the DNS failure was temporary, this will allow the
>
[jira] [Commented] (HADOOP-12897) KerberosAuthenticator.authenticate to include URL on IO failures
[
https://issues.apache.org/jira/browse/HADOOP-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361272#comment-16361272
]
Arpit Agarwal commented on HADOOP-12897:
The v5 patch looks good. One minor comment. We need spaces before the opening
braces, not sure why checkstyle didn't flag this.
{code}
} catch (IOException ex){
{code}
[~ste...@apache.org], any further comments?
> KerberosAuthenticator.authenticate to include URL on IO failures
>
>
> Key: HADOOP-12897
> URL: https://issues.apache.org/jira/browse/HADOOP-12897
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Ajay Kumar
>Priority: Minor
> Attachments: HADOOP-12897.001.patch, HADOOP-12897.002.patch,
> HADOOP-12897.003.patch, HADOOP-12897.004.patch, HADOOP-12897.005.patch
>
>
> If {{KerberosAuthenticator.authenticate}} can't connect to the endpoint, you
> get a stack trace, but without the URL it is trying to talk to.
> That is: it doesn't have any equivalent of the {{NetUtils.wrapException}}
> handler —which can't be called here as its not in the {{hadoop-auth}} module
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13551) hook up AwsSdkMetrics to hadoop metrics
[
https://issues.apache.org/jira/browse/HADOOP-13551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361241#comment-16361241
]
Sean Mackrory commented on HADOOP-13551:
I hadn't started on it, no. Do you know what kind of timeframe 3.1.0 is aiming
for?
> hook up AwsSdkMetrics to hadoop metrics
> ---
>
> Key: HADOOP-13551
> URL: https://issues.apache.org/jira/browse/HADOOP-13551
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Sean Mackrory
>Priority: Critical
>
> There's an API in {{com.amazonaws.metrics.AwsSdkMetrics}} to give access to
> the internal metrics of the AWS libraries. We might want to get at those
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15139) [Umbrella] Improvements and fixes for Hadoop shaded client work
[ https://issues.apache.org/jira/browse/HADOOP-15139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361199#comment-16361199 ] Bharat Viswanadham commented on HADOOP-15139: - Hi [~leftnoteasy] For this Umbrella Jira, only opened task as of now is HADOOP-15137, I have provided patch for it. Waiting for review. Could you help in reviewing the patch? > [Umbrella] Improvements and fixes for Hadoop shaded client work > > > Key: HADOOP-15139 > URL: https://issues.apache.org/jira/browse/HADOOP-15139 > Project: Hadoop Common > Issue Type: Bug >Reporter: Junping Du >Assignee: Bharat Viswanadham >Priority: Critical > > In HADOOP-11656, we have made great progress in splitting out third-party > dependencies from shaded hadoop client jar (hadoop-client-api), put runtime > dependencies in hadoop-client-runtime, and have shaded version of > hadoop-client-minicluster for test. However, there are still some left work > for this feature to be fully completed: > - We don't have a comprehensive documentation to guide downstream > projects/users to use shaded JARs instead of previous JARs > - We should consider to wrap up hadoop tools (distcp, aws, azure) to have > shaded version > - More issues could be identified when shaded jars are adopted in more test > and production environment, like HADOOP-15137. > Let's have this umbrella JIRA to track all efforts that left to improve > hadoop shaded client effort. > CC [~busbey], [~bharatviswa] and [~vinodkv]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14961) Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed
[
https://issues.apache.org/jira/browse/HADOOP-14961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HADOOP-14961:
-
Committer of this patch, please set the *Fix Version/s* for this JIRA.
> Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed
> --
>
> Key: HADOOP-14961
> URL: https://issues.apache.org/jira/browse/HADOOP-14961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, test
>Affects Versions: 3.1.0
>Reporter: John Zhuge
>Priority: Major
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/13546/console
> {noformat}
> Downloading Oracle Java 8...
> [0m[91m--2017-10-18 18:28:11--
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
>
> [0m[91mResolving download.oracle.com (download.oracle.com)...
> [0m[91m23.59.190.131, 23.59.190.130
> Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80...
> [0m[91mconnected.
> HTTP request sent, awaiting response... [0m[91m302 Moved Temporarily
> Location:
> https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
> [following]
> --2017-10-18 18:28:11--
> https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
>
> [0m[91mResolving edelivery.oracle.com (edelivery.oracle.com)...
> [0m[91m23.39.16.136, 2600:1409:a:39c::2d3e, 2600:1409:a:39e::2d3e
> Connecting to edelivery.oracle.com
> (edelivery.oracle.com)|23.39.16.136|:443... [0m[91mconnected.
> [0m[91mHTTP request sent, awaiting response... [0m[91m302 Moved
> Temporarily
> Location:
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c
> [following]
> --2017-10-18 18:28:11--
> http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c
>
> Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80...
> [0m[91mconnected.
> HTTP request sent, awaiting response... [0m[91m404 Not Found
> [0m[91m2017-10-18 18:28:12 ERROR 404: Not Found.
> [0m[91mdownload failed
> Oracle JDK 8 is NOT installed.
> {noformat}
> Looks like Oracle JDK 8u144 is no longer available for download using that
> link. 8u151 and 8u152 are available.
> Many of last 10 https://builds.apache.org/job/PreCommit-HADOOP-Build/ jobs
> failed the same way, all on build host H1 and H6.
> [~aw] has a patch available in HADOOP-14816 "Update Dockerfile to use Xenial"
> for a long term fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15221) Swift driver should not fail if JSONUtils reports UnknowPropertyException
Chen He created HADOOP-15221: Summary: Swift driver should not fail if JSONUtils reports UnknowPropertyException Key: HADOOP-15221 URL: https://issues.apache.org/jira/browse/HADOOP-15221 Project: Hadoop Common Issue Type: Improvement Components: fs/swift Reporter: Chen He Assignee: Chen He org.apache.hadoop.fs.swift.exceptions.SwiftJsonMarshallingException: org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized field We know system is keep involving and new field will be added. However, for compatibility point of view, extra field added to json should be logged but may not lead to failure from the robustness point of view. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15040) Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation
[
https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361090#comment-16361090
]
Steve Loughran commented on HADOOP-15040:
-
Tested s3 ireland, s3guard on, all well
+1
> Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation
> ---
>
> Key: HADOOP-15040
> URL: https://issues.apache.org/jira/browse/HADOOP-15040
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
>Priority: Blocker
> Attachments: HADOOP-15040.001.patch
>
>
> My colleagues working with Yarn log aggregation found that they were getting
> this message spammed in their logs when they used an s3a:// URI for logs
> (yarn.nodemanager.remote-app-log-dir):
> {noformat}
> getting attribute Region of com.amazonaws.management:type=AwsSdkMetrics threw
> an exception
> javax.management.RuntimeMBeanException: java.lang.NullPointerException
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
> at
>
> Caused by: java.lang.NullPointerException
> at com.amazonaws.metrics.AwsSdkMetrics.getRegion(AwsSdkMetrics.java:729)
> at com.amazonaws.metrics.MetricAdmin.getRegion(MetricAdmin.java:67)
> at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> {noformat}
> This happens even though the aws sdk cloudwatch metrics reporting was
> disabled (default), which is a bug.
> I filed a [github issue|https://github.com/aws/aws-sdk-java/issues/1375|] and
> it looks like a fix should be coming around SDK release 1.11.229 or so.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14603) S3A input stream to support ByteBufferReadable
[
https://issues.apache.org/jira/browse/HADOOP-14603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360973#comment-16360973
]
Keith Godwin Chapman commented on HADOOP-14603:
---
[~ste...@apache.org] Do you have any hints on how this can be implemented? I'd
be happy o implement it and submit as a patch. I've looked at the amazon S3 SDK
and did not see any read methods that support ByteBuffer in it's API.
> S3A input stream to support ByteBufferReadable
> --
>
> Key: HADOOP-14603
> URL: https://issues.apache.org/jira/browse/HADOOP-14603
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Priority: Minor
>
> S3AInputStream could support {{ByteBufferReadable,
> HasEnhancedByteBufferAccess}} and the operations to read into byte buffers.
> This is only if we can see a clear performance benefit from doing this or the
> API is being more broadly used
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14831) Über-jira: S3a phase IV: Hadoop 3.1 features
[ https://issues.apache.org/jira/browse/HADOOP-14831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360946#comment-16360946 ] Wangda Tan commented on HADOOP-14831: - [~ste...@apache.org], thanks for splitting 3.1.0 and 3.2.0 works. I just quickly scanned these tickets, I found there're 1 open blocker (PA) and 3 open criticals (without patch). [1]. Is there any timeline to finish these tasks? If these tasks cannot be done by Feb 18 (which is our proposed time to do first RC and start vote), can I move them to 3.2.0? [1] parent = HADOOP-14831 AND priority in (critical, blocker) ORDER BY priority DESC > Über-jira: S3a phase IV: Hadoop 3.1 features > > > Key: HADOOP-14831 > URL: https://issues.apache.org/jira/browse/HADOOP-14831 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Priority: Major > > All the S3/S3A features targeting Hadoop 3.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14831) Über-jira: S3a phase IV: Hadoop 3.1 features
[ https://issues.apache.org/jira/browse/HADOOP-14831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360907#comment-16360907 ] Steve Loughran commented on HADOOP-14831: - I've just created a Hadoop 3.2 Uber JIRA (HADOOP-15520) and moved (there's now a bulk move!) those issues which were non-critical features that don't have much/any activity or quick solutions. Left here: important things I'd like to see in 3.1; issues with ready to review/refresh/apply patches, test additions (low risk), and issues which seem more bugs than features > Über-jira: S3a phase IV: Hadoop 3.1 features > > > Key: HADOOP-14831 > URL: https://issues.apache.org/jira/browse/HADOOP-14831 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Priority: Major > > All the S3/S3A features targeting Hadoop 3.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15040) Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation
[
https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15040:
Summary: Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log
Aggregation (was: Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation)
> Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation
> ---
>
> Key: HADOOP-15040
> URL: https://issues.apache.org/jira/browse/HADOOP-15040
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
>Priority: Blocker
> Attachments: HADOOP-15040.001.patch
>
>
> My colleagues working with Yarn log aggregation found that they were getting
> this message spammed in their logs when they used an s3a:// URI for logs
> (yarn.nodemanager.remote-app-log-dir):
> {noformat}
> getting attribute Region of com.amazonaws.management:type=AwsSdkMetrics threw
> an exception
> javax.management.RuntimeMBeanException: java.lang.NullPointerException
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
> at
>
> Caused by: java.lang.NullPointerException
> at com.amazonaws.metrics.AwsSdkMetrics.getRegion(AwsSdkMetrics.java:729)
> at com.amazonaws.metrics.MetricAdmin.getRegion(MetricAdmin.java:67)
> at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> {noformat}
> This happens even though the aws sdk cloudwatch metrics reporting was
> disabled (default), which is a bug.
> I filed a [github issue|https://github.com/aws/aws-sdk-java/issues/1375|] and
> it looks like a fix should be coming around SDK release 1.11.229 or so.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15187) Remove ADL mock test dependency on REST call invoked from Java SDK
[ https://issues.apache.org/jira/browse/HADOOP-15187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360889#comment-16360889 ] Hudson commented on HADOOP-15187: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13644 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13644/]) HADOOP-15187. Remove ADL mock test dependency on REST call invoked from (stevel: rev 8cf88fcd1f63d3d4e9736b1b687a4f4e663f6125) * (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestAdlRead.java * (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestListStatus.java * (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestConcurrentDataReadOperations.java * (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestableAdlFileSystem.java * (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/AdlMockWebServer.java * (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestGetFileStatus.java * (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/common/TestDataForRead.java * (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestACLFeatures.java * (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/common/ExpectedResponse.java * (delete) hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestCustomTokenProvider.java > Remove ADL mock test dependency on REST call invoked from Java SDK > --- > > Key: HADOOP-15187 > URL: https://issues.apache.org/jira/browse/HADOOP-15187 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0 >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-15187-001.patch > > > Cleanup unit test which mocks REST calls invoked within dependency SDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[
https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360887#comment-16360887
]
genericqa commented on HADOOP-14661:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-14661 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14661 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12877218/HADOOP-14661.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14101/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3A to support Requester Pays Buckets
> -
>
> Key: HADOOP-14661
> URL: https://issues.apache.org/jira/browse/HADOOP-14661
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common, util
>Affects Versions: 3.0.0-alpha3
>Reporter: Mandus Momberg
>Assignee: Mandus Momberg
>Priority: Minor
> Attachments: HADOOP-14661.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Amazon S3 has the ability to charge the requester for the cost of accessing
> S3. This is called Requester Pays Buckets.
> In order to access these buckets, each request needs to be signed with a
> specific header.
> http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14975) S3AInputStream/OutputStream statistics aren't getting into StorageStatistics
[ https://issues.apache.org/jira/browse/HADOOP-14975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-14975. - Resolution: Duplicate Duplicate of HADOOP-15161; that is fixed so closing this one > S3AInputStream/OutputStream statistics aren't getting into StorageStatistics > > > Key: HADOOP-14975 > URL: https://issues.apache.org/jira/browse/HADOOP-14975 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > > when the input and output stream stats are merged into the > S3AInstrumentation, the FS statistics aren't updated to match, so FS > statistics don't track things like aggregate throttle count, TCP aborts, > bytes discarded etc. They are metrics, but not sotrage stats > They should be, which requires S3AInstrumentation to take the StorageStats in > its constructor and then update. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13525) Optimize uses of FS operations in the ASF analysis frameworks and libraries
[
https://issues.apache.org/jira/browse/HADOOP-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13525:
Parent: HADOOP-15220 (was: HADOOP-14831)
> Optimize uses of FS operations in the ASF analysis frameworks and libraries
>
>
> Key: HADOOP-13525
> URL: https://issues.apache.org/jira/browse/HADOOP-13525
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> Review uses of the FS APIs in applications using the Hadoop FS API to access
> filesystems; identify suboptimal uses and tune them for better performance
> against HDFS and object stores
> * Assume arbitrary Hadoop 2.x releases: make no changes which are known to
> make operations on older versions of Hadoop slower
> * Do propose those changes which deliver speedups in later versions of
> Hadoop, while not impacting older versions, or risk of causing scalability
> problems.
> * Add more tests, especially scalable ones which also display metrics.
> * Use standard benchmarks and optimization tools to identify hotspots.
> * Use FS behaviour as verified in the FS contract tests as evidence that
> filesystems correctly implement the Hadoop FS APIs. If a use of an API call
> is made which hints at the expectation of different/untested behaviours,
> leave alone and add new tests to the Hadoop FS contract to determine cross-FS
> semantics.
> * Focus on the startup, split calculation and directory scanning operations:
> the ones which slow down entire queries.
> * Eliminate use of {{isDirectory()}}, {{getLength()}}, {{exists()}} if a
> followon operation ({{getStatus()}},{{delete()}}, ... makes the use redundant.
> * Assume that {{FileStatus}} entries are not cached; the cost of creating
> them is 1 RPC call against HDFS, 1+ HTTPS call against object stores.
> * Locate calls to the listing operations, identify speedups, especially on
> recursive directory scans.
> * Identify suboptimal seek patterns (backwards as well as forwards) and
> attempt to reduce/eliminate through reordering and result caching.
> * Try to reuse the results of previous operations (e.g {{FileStatus}}
> instances) in follow-on calls.
> * Commonly used file formats (e.g ORC) will have transitive benefits.
> * Frameworks to use predicate pushdown where this delivers speedups
> * Document best practises identified and implemented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14661: Parent: HADOOP-15220 (was: HADOOP-14831) > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14833) Move s3a user:secret auth out of the main chain of auth mechanisms
[ https://issues.apache.org/jira/browse/HADOOP-14833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14833: Parent: HADOOP-15220 (was: HADOOP-14831) > Move s3a user:secret auth out of the main chain of auth mechanisms > -- > > Key: HADOOP-14833 > URL: https://issues.apache.org/jira/browse/HADOOP-14833 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Priority: Minor > > Remove the s3a://user:secret@host auth mechanism from S3a > I think we could consider retain it as an explicit credential provider you > can ask for, so that people who cannot move off it (yet) can reconfigure > their system, but unless you do that, it stops working. > We could add a dummy credential handler which recognises the user:secret > pattern & then tells the user "no longer supported, sorry, here's how to > migrate", & add that to the default chain after everything else. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13853) S3ADataBlocks.DiskBlock to lazy create dest file for faster 0-byte puts
[
https://issues.apache.org/jira/browse/HADOOP-13853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13853:
Parent: HADOOP-15220 (was: HADOOP-14831)
> S3ADataBlocks.DiskBlock to lazy create dest file for faster 0-byte puts
> ---
>
> Key: HADOOP-13853
> URL: https://issues.apache.org/jira/browse/HADOOP-13853
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
>
> Looking at traces of work, there's invariably a PUT of a _SUCCESS at the end,
> which, with disk output, adds the overhead of creating, writing to and then
> reading a 0 byte file.
> With a lazy create, the creation could be postponed until the first write,
> with special handling in the {{startUpload()}} operation to return a null
> stream, rather than reopen the file. Saves on some disk IO: create, read,
> delete
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13330) s3a large subdir delete to do listObjects() and delete() calls in parallel
[ https://issues.apache.org/jira/browse/HADOOP-13330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13330: Parent: HADOOP-15220 (was: HADOOP-14831) > s3a large subdir delete to do listObjects() and delete() calls in parallel > -- > > Key: HADOOP-13330 > URL: https://issues.apache.org/jira/browse/HADOOP-13330 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > When doing deletes of large directories/directory trees, paged list+delete > calls will be made. It would be faster if the delete and list calls were done > in parallel; the next listing batch obtained while the current batch was > being deleted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13007) cherry pick s3 ehancements from PrestoS3FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-13007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13007: Parent: HADOOP-15220 (was: HADOOP-14831) > cherry pick s3 ehancements from PrestoS3FileSystem > -- > > Key: HADOOP-13007 > URL: https://issues.apache.org/jira/browse/HADOOP-13007 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > > Looking at > [[https://github.com/prestodb/presto/blob/master/presto-hive/src/main/java/com/facebook/presto/hive/PrestoS3FileSystem.java]], > they've done some interesting things: configurable connection timeouts and, > retry options, statistics to count exceptions caught/re-opened, and more > review them, if there is good stuff there, add it to S3a -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15105) add htrace context to HTTP requests as ? parameter
[ https://issues.apache.org/jira/browse/HADOOP-15105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15105: Parent: HADOOP-15220 (was: HADOOP-14831) > add htrace context to HTTP requests as ? parameter > -- > > Key: HADOOP-15105 > URL: https://issues.apache.org/jira/browse/HADOOP-15105 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Priority: Minor > > you can [add x-something query parameters to S3 REST > calls|http://docs.aws.amazon.com/AmazonS3/latest/dev/LogFormat.html] > These then get included in the logs. If the htrace context were passed in > this way, you could determine from the S3 logs what query/job an HTTP request > ties back to -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14556:
Parent: HADOOP-15220 (was: HADOOP-14831)
> S3A to support Delegation Tokens
>
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13507) export s3a BlockingThreadPoolExecutorService pool info (size, load) as metrics
[
https://issues.apache.org/jira/browse/HADOOP-13507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13507:
Parent: HADOOP-15220 (was: HADOOP-14831)
> export s3a BlockingThreadPoolExecutorService pool info (size, load) as metrics
> --
>
> Key: HADOOP-13507
> URL: https://issues.apache.org/jira/browse/HADOOP-13507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
>
> We should publish load info on {{BlockingThreadPoolExecutorService}} as s3a
> metrics: size, available, maybe even some timer info on load (at least: rate
> of recent semaphore acquire/release)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14093) add tests/docs for HAR files on s3a
[ https://issues.apache.org/jira/browse/HADOOP-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14093: Parent: HADOOP-15220 (was: HADOOP-14831) > add tests/docs for HAR files on s3a > --- > > Key: HADOOP-14093 > URL: https://issues.apache.org/jira/browse/HADOOP-14093 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3, test >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > Because many small files are such a perf killer for S3, we could think about > encouraging people to use HAR files where they need to. Which means: tests > and docs for using this + S3a -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13611) FileSystem/s3a processDeleteOnExit to skip the exists() check
[
https://issues.apache.org/jira/browse/HADOOP-13611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13611:
Parent: HADOOP-15220 (was: HADOOP-14831)
> FileSystem/s3a processDeleteOnExit to skip the exists() check
> -
>
> Key: HADOOP-13611
> URL: https://issues.apache.org/jira/browse/HADOOP-13611
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/s3
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Priority: Trivial
>
> If you look at {{FileSystem.processDeleteOnExit()}}, it does an exists()
> check for each entry, before calling delete().
> That exists() check is superfluous; on s3 it add an extra 1-4 HTTP GETs
> This could be fixed with a subclass in s3a to avoid it, but as the call is
> superfluous in *all* filesystems, it could be removed in {{FileSystem}} and
> so picked up by all object stores.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13704) S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use
[
https://issues.apache.org/jira/browse/HADOOP-13704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13704:
Parent: HADOOP-15220 (was: HADOOP-14831)
> S3A getContentSummary() to move to listFiles(recursive) to count children;
> instrument use
> -
>
> Key: HADOOP-13704
> URL: https://issues.apache.org/jira/browse/HADOOP-13704
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
>
> Hive and a bit of Spark use {{getContentSummary()}} to get some summary stats
> of a filesystem. This is very expensive on S3A (and any other object store),
> especially as the base implementation does the recursive tree walk.
> Because of HADOOP-13208, we have a full enumeration of files under a path
> without directory costs...S3A can/should switch to this to speed up those
> places where the operation is called.
> Also
> * API call needs FS spec and contract tests
> * S3A could instrument invocation, so as to enable real-world popularity to
> be measured
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13846) S3A to implement rename(final Path src, final Path dst, final Rename... options)
[
https://issues.apache.org/jira/browse/HADOOP-13846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13846:
Parent: HADOOP-15220 (was: HADOOP-14831)
> S3A to implement rename(final Path src, final Path dst, final Rename...
> options)
>
>
> Key: HADOOP-13846
> URL: https://issues.apache.org/jira/browse/HADOOP-13846
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Major
>
> S3a now raises exceptions on invalid rename operations, but these get lost. I
> plan to use them in my s3guard committer HADOOP-13786.
> Rather than just make innerRename() private, S3A could implement
> {{FileSystem.rename(final Path src, final Path dst, final Rename...
> options)}} and so have an exception-raising rename which can be called
> without going more into the internals.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13293) add a special 0 byte input stream for empty blobs
[ https://issues.apache.org/jira/browse/HADOOP-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13293: Parent: HADOOP-15220 (was: HADOOP-14831) > add a special 0 byte input stream for empty blobs > - > > Key: HADOOP-13293 > URL: https://issues.apache.org/jira/browse/HADOOP-13293 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > S3a (and the other object stores) have a lot of IO going on, even for 0 byte > files. They don't need to: that's a special case which can be handled > locally. A special ZeroByteInputStream class could handle this for all the > object stores. > This isn't much of an optimization: code shouldn't normally need to go > through 0 byte files, but we see evidence it does sometimes happen. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13892) use s3 tags/headers to record permissions on objects, so preserving them through distcp round trips
[ https://issues.apache.org/jira/browse/HADOOP-13892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13892: Parent: HADOOP-15220 (was: HADOOP-14831) > use s3 tags/headers to record permissions on objects, so preserving them > through distcp round trips > --- > > Key: HADOOP-13892 > URL: https://issues.apache.org/jira/browse/HADOOP-13892 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Major > > S3 now supports object tags, attributes which can be updated during the life > of an object. > S3A could use that to preserve the permissions/ACLs of objects when copied > from elsewhere, in particular from HDFS. This would ensure that data backed > up from HDFS preserves all the permission information needed when doing a > recovery from S3 to HDFS. > Azure WASB does exactly this already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13311) S3A shell entry point to support commands specific to S3A.
[
https://issues.apache.org/jira/browse/HADOOP-13311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13311:
Parent: HADOOP-15220 (was: HADOOP-14831)
> S3A shell entry point to support commands specific to S3A.
> --
>
> Key: HADOOP-13311
> URL: https://issues.apache.org/jira/browse/HADOOP-13311
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Chris Nauroth
>Priority: Minor
>
> Create a new {{s3a}} shell entry point. This can support diagnostic and
> administrative commands that are specific to S3A and wouldn't make sense to
> group under existing scripts like {{hadoop}} or {{hdfs}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14132) Filesystem discovery to stop loading implementation classes
[ https://issues.apache.org/jira/browse/HADOOP-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14132: Parent: HADOOP-15220 (was: HADOOP-14831) > Filesystem discovery to stop loading implementation classes > --- > > Key: HADOOP-14132 > URL: https://issues.apache.org/jira/browse/HADOOP-14132 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/adl, fs/azure, fs/oss, fs/s3, fs/swift >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Integration testing of Hadoop with the HADOOP-14040 has shown up that the > move to a shaded AWS JAR is slowing all hadoop client code down. > I believe this is due to how we use service discovery to identify FS > implementations: the implementation classes themselves are instantiated. > This has known problems today with classloading, but clearly impacts > performance too, especially with complex transitive dependencies unique to > the loaded class. > Proposed: have lightweight service declaration classes which implement an > interface declaring > # schema > # classname of FileSystem impl > # classname of AbstractFS impl > # homepage (for third party code, support, etc) > These are what we register and scan in the FS to look for services. > This will leave the question about what to do for existing filesystems? I > think we'll need to retain the old code for external ones, while moving the > hadoop modules to the new ones -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13600) S3a rename() to copy files in a directory in parallel
[ https://issues.apache.org/jira/browse/HADOOP-13600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13600: Parent: HADOOP-15220 (was: HADOOP-14831) > S3a rename() to copy files in a directory in parallel > - > > Key: HADOOP-13600 > URL: https://issues.apache.org/jira/browse/HADOOP-13600 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Sahil Takiar >Priority: Major > Attachments: HADOOP-13600.001.patch > > > Currently a directory rename does a one-by-one copy, making the request > O(files * data). If the copy operations were launched in parallel, the > duration of the copy may be reducable to the duration of the longest copy. > For a directory with many files, this will be significant -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15201) Automatically determine region & hence S3 endpoint of buckets
[
https://issues.apache.org/jira/browse/HADOOP-15201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15201:
Parent: HADOOP-15220 (was: HADOOP-14831)
> Automatically determine region & hence S3 endpoint of buckets
> -
>
> Key: HADOOP-15201
> URL: https://issues.apache.org/jira/browse/HADOOP-15201
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Priority: Minor
>
> Use {{getBucketLocation}} to determine location & map to endpoint, if
> fs.s3a.endpoint is set to "automatic"
> S3guard added the API call {{String getBucketLocation()}}, which is used for
> DDB binding, We can also use this to determine the s3 endpoint, to avoid
> recurrent issues with auth failures related to it not being valid
> Still need to handle: buckets on third-party servers, and the inevitability
> that new AWS regions will be added after the Hadoop version & AWS Jar is
> shipped and frozen
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14837) Handle S3A "glacier" data
[ https://issues.apache.org/jira/browse/HADOOP-14837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14837: Parent: HADOOP-15220 (was: HADOOP-14831) > Handle S3A "glacier" data > - > > Key: HADOOP-14837 > URL: https://issues.apache.org/jira/browse/HADOOP-14837 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Priority: Minor > > SPARK-21797 covers how if you have AWS S3 set to copy some files to glacier, > they appear in the listing but GETs fail, and so does everything else > We should think about how best to handle this. > # report better > # if listings can identify files which are glaciated then maybe we could have > an option to filter them out > # test & see what happens -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15135) S3a to support get/set permissions through S3 object tags
[ https://issues.apache.org/jira/browse/HADOOP-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15135: Parent: HADOOP-15220 (was: HADOOP-14831) > S3a to support get/set permissions through S3 object tags > - > > Key: HADOOP-15135 > URL: https://issues.apache.org/jira/browse/HADOOP-15135 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran >Priority: Major > > Azure wasb supports get/set permissions (for persistence only) to help round > trip distcp operations. > Could aws tags be used similarly? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14603) S3A input stream to support ByteBufferReadable
[
https://issues.apache.org/jira/browse/HADOOP-14603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14603:
Parent: HADOOP-15220 (was: HADOOP-14831)
> S3A input stream to support ByteBufferReadable
> --
>
> Key: HADOOP-14603
> URL: https://issues.apache.org/jira/browse/HADOOP-14603
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Priority: Minor
>
> S3AInputStream could support {{ByteBufferReadable,
> HasEnhancedByteBufferAccess}} and the operations to read into byte buffers.
> This is only if we can see a clear performance benefit from doing this or the
> API is being more broadly used
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13695) S3A to use a thread pool for async path operations
[
https://issues.apache.org/jira/browse/HADOOP-13695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13695:
Parent: HADOOP-15220 (was: HADOOP-14831)
> S3A to use a thread pool for async path operations
> --
>
> Key: HADOOP-13695
> URL: https://issues.apache.org/jira/browse/HADOOP-13695
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Major
>
> S3A path operations are often slow due to directory scanning, mock directory
> create/delete, etc. Many of these can be done asynchronously
> * because deletion is eventually consistent, deleting parent dirs after an
> operation has returned doesn't alter the behaviour, except in the special
> case of : operation failure.
> * scanning for paths/parents of a file in the create operation only needs to
> complete before the close() operation instantiates the object, no need to
> block create().
> * parallelized COPY calls would permit asynchronous rename.
> We could either use the thread pool used for block writes, or somehow isolate
> low cost path ops (GET, DELETE) from the more expensive calls (COPY, PUT) so
> that a thread doing basic IO doesn't block for the duration of the long op.
> Maybe also use {{Semaphore.tryAcquire()}} and only start async work if there
> actually is an idle thread, doing it synchronously if not. Maybe it depends
> on the operation. path query/cleanup before/after a write is something which
> could be scheduled as just more futures to schedule in the block write.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13407) s3a directory housekeeping operations to be done in async thread
[ https://issues.apache.org/jira/browse/HADOOP-13407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13407: Parent: HADOOP-15220 (was: HADOOP-14831) > s3a directory housekeeping operations to be done in async thread > > > Key: HADOOP-13407 > URL: https://issues.apache.org/jira/browse/HADOOP-13407 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > Some of the delays on s3a calls are due to cleaning up parent pseudo > directories; repeated getParent/GET calls to look for the entries, then to > delete them. > We could possibly make this asynchronous; the core semantics would be > retained, just the cleanup delayed. > Risks? > # while the cleanup is in progress, getFileStatus of parent dirs could imply > that the parent dir is still empty > # failure > of course, these risks exist today. We really need an s3a fsck -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12020) Support AWS S3 reduced redundancy storage class
[
https://issues.apache.org/jira/browse/HADOOP-12020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-12020:
Parent: HADOOP-15220 (was: HADOOP-14831)
> Support AWS S3 reduced redundancy storage class
> ---
>
> Key: HADOOP-12020
> URL: https://issues.apache.org/jira/browse/HADOOP-12020
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.0
> Environment: Hadoop on AWS
>Reporter: Yann Landrin-Schweitzer
>Priority: Major
>
> Amazon S3 uses, by default, the NORMAL_STORAGE class for s3 objects.
> This offers, according to Amazon's material, 99.% reliability.
> For many applications, however, the 99.99% reliability offered by the
> REDUCED_REDUNDANCY storage class is amply sufficient, and comes with a
> significant cost saving.
> HDFS, when using the legacy s3n protocol, or the new s3a scheme, should
> support overriding the default storage class of created s3 objects so that
> users can take advantage of this cost benefit.
> This would require minor changes of the s3n and s3a drivers, using
> a configuration property fs.s3n.storage.class to override the default storage
> when desirable.
> This override could be implemented in Jets3tNativeFileSystemStore with:
> S3Object object = new S3Object(key);
> ...
> if(storageClass!=null) object.setStorageClass(storageClass);
> It would take a more complex form in s3a, e.g. setting:
> InitiateMultipartUploadRequest initiateMPURequest =
> new InitiateMultipartUploadRequest(bucket, key, om);
> if(storageClass !=null ) {
> initiateMPURequest =
> initiateMPURequest.withStorageClass(storageClass);
> }
> and similar statements in various places.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13432) S3A: Consider using TransferManager.download for copyToLocalFile
[ https://issues.apache.org/jira/browse/HADOOP-13432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13432: Parent: HADOOP-15220 (was: HADOOP-14831) > S3A: Consider using TransferManager.download for copyToLocalFile > > > Key: HADOOP-13432 > URL: https://issues.apache.org/jira/browse/HADOOP-13432 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Rajesh Balamohan >Priority: Minor > > Currently it relies on the default implementation of FileSystem. But it would > be good to explore using TransferManager.download() (Ref: > https://java.awsblog.com/post/Tx3Z7NO7C2TVLB/Parallelizing-Large-Downloads-for-Optimal-Speed > for recent aws-sdk-java). When aws-sdk version is bumped it,it would > automatically get the benefit of parallel download sas well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14747) S3AInputStream to implement CanUnbuffer
[
https://issues.apache.org/jira/browse/HADOOP-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14747:
Parent: HADOOP-15220 (was: HADOOP-14831)
> S3AInputStream to implement CanUnbuffer
> ---
>
> Key: HADOOP-14747
> URL: https://issues.apache.org/jira/browse/HADOOP-14747
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Priority: Major
>
> HBase relies on FileSystems implementing {{CanUnbuffer.unbuffer()}} to force
> input streams to free up remote connections (HBASE-9393). This works for
> HDFS, but not elsewhere.
> S3A input stream can implement {{CanUnbuffer.unbuffer()}} by closing the
> input stream and relying on lazy seek to reopen it on demand.
> Needs
> * Contract specification of unbuffer. As in "who added a new feature to
> filesystems but forgot to mention what it should do?"
> * Contract test for filesystems which declare their support.
> * S3AInputStream to call {{closeStream()}} on a call to {{unbuffer()}}.
> * Test case
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13974) S3Guard CLI to support list/purge of pending multipart commits
[ https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-13974. - Resolution: Fixed > S3Guard CLI to support list/purge of pending multipart commits > -- > > Key: HADOOP-13974 > URL: https://issues.apache.org/jira/browse/HADOOP-13974 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Aaron Fabbri >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-13974.001.patch, HADOOP-13974.002.patch, > HADOOP-13974.003.patch, HADOOP-13974.004.patch, HADOOP-13974.005.patch, > HADOOP-13974.006.patch, HADOOP-13974.007.patch > > > The S3A CLI will need to be able to list and delete pending multipart > commits. > We can do the cleanup already via fs.s3a properties. The CLI will let scripts > stat for outstanding data (have a different exit code) and permit batch jobs > to explicitly trigger cleanups. > This will become critical with the multipart committer, as there's a > significantly higher likelihood of commits remaining outstanding. > We may also want to be able to enumerate/cancel all pending commits in the FS > tree -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15187) Remove ADL mock test dependency on REST call invoked from Java SDK
[ https://issues.apache.org/jira/browse/HADOOP-15187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15187: Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) LGTM +1, committed to hadoop 3.1. Thanks! > Remove ADL mock test dependency on REST call invoked from Java SDK > --- > > Key: HADOOP-15187 > URL: https://issues.apache.org/jira/browse/HADOOP-15187 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0 >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-15187-001.patch > > > Cleanup unit test which mocks REST calls invoked within dependency SDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15187) Remove ADL mock test dependency on REST call invoked from Java SDK
[ https://issues.apache.org/jira/browse/HADOOP-15187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15187: Summary: Remove ADL mock test dependency on REST call invoked from Java SDK (was: Remove mock test dependency on REST call invoked from Java SDK ) > Remove ADL mock test dependency on REST call invoked from Java SDK > --- > > Key: HADOOP-15187 > URL: https://issues.apache.org/jira/browse/HADOOP-15187 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0 >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane >Priority: Major > Attachments: HADOOP-15187-001.patch > > > Cleanup unit test which mocks REST calls invoked within dependency SDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12949) Add HTrace to the s3a connector
[ https://issues.apache.org/jira/browse/HADOOP-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-12949: Target Version/s: 3.2.0 (was: 3.1.0) pushing out 3.2, but still hoping to see some patches here. Instrumenting the filesystems will be the motivator for instrumenting the layers above > Add HTrace to the s3a connector > --- > > Key: HADOOP-12949 > URL: https://issues.apache.org/jira/browse/HADOOP-12949 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Madhawa Gunasekara >Assignee: Madhawa Gunasekara >Priority: Major > > Hi All, > s3, GCS, WASB, and other cloud blob stores are becoming increasingly > important in Hadoop. But we don't have distributed tracing for these yet. It > would be interesting to add distributed tracing here. It would enable > collecting really interesting data like probability distributions of PUT and > GET requests to s3 and their impact on MR jobs, etc. > I would like to implement this feature, Please shed some light on this > Thanks, > Madhawa -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12949) Add HTrace to the s3a connector
[ https://issues.apache.org/jira/browse/HADOOP-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-12949: Parent Issue: HADOOP-15220 (was: HADOOP-14831) > Add HTrace to the s3a connector > --- > > Key: HADOOP-12949 > URL: https://issues.apache.org/jira/browse/HADOOP-12949 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Madhawa Gunasekara >Assignee: Madhawa Gunasekara >Priority: Major > > Hi All, > s3, GCS, WASB, and other cloud blob stores are becoming increasingly > important in Hadoop. But we don't have distributed tracing for these yet. It > would be interesting to add distributed tracing here. It would enable > collecting really interesting data like probability distributions of PUT and > GET requests to s3 and their impact on MR jobs, etc. > I would like to implement this feature, Please shed some light on this > Thanks, > Madhawa -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14439) regression: secret stripping from S3x URIs breaks some downstream code
[
https://issues.apache.org/jira/browse/HADOOP-14439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360747#comment-16360747
]
Steve Loughran commented on HADOOP-14439:
-
[~vinayrpet] reviewing this, we've not had anyone complaining for a while, and
given the security implications of putting secrets in URIs, that makes me happy.
how about we close this as a WONTFIX?
> regression: secret stripping from S3x URIs breaks some downstream code
> --
>
> Key: HADOOP-14439
> URL: https://issues.apache.org/jira/browse/HADOOP-14439
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: Spark 2.1
>Reporter: Steve Loughran
>Assignee: Vinayakumar B
>Priority: Minor
> Attachments: HADOOP-14439-01.patch, HADOOP-14439-02.patch
>
>
> Surfaced in SPARK-20799
> Spark is listing the contents of a path with getFileStatus(path), then
> looking up the path value doing a lookup of the contents.
> Apparently the lookup is failing to find files if you have a secret in the
> key, {{s3a://key:secret@bucket/path}}.
> Presumably this is because the stripped values aren't matching.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret,key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14507:
Priority: Critical (was: Major)
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret,key
>
>
> Key: HADOOP-14507
> URL: https://issues.apache.org/jira/browse/HADOOP-14507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-14507-001.patch, HADOOP-14507-002.patch,
> HADOOP-14507-003.patch, HADOOP-14507-004.patch, HADOOP-14507-005.patch,
> HADOOP-14507-006.patch, HADOOP-14507-006.patch
>
>
> Per-bucket jceks support turns out to be complex as you have to manage
> multiple jecks files & configure the client to ask for the right one. This is
> because we're calling {{Configuration.getPassword{"fs,s3a.secret.key"}}.
> If before that, we do a check for the explict id, key, session key in the
> properties {{fs.s3a.$bucket.secret}} ( & c), we could have a single JCEKs
> file with all the secrets for different bucket. You would only need to
> explicitly point the base config to the secrets file, and the right
> credentials would be picked up, if set
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13892) use s3 tags/headers to record permissions on objects, so preserving them through distcp round trips
[ https://issues.apache.org/jira/browse/HADOOP-13892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-13892. - Resolution: Won't Fix > use s3 tags/headers to record permissions on objects, so preserving them > through distcp round trips > --- > > Key: HADOOP-13892 > URL: https://issues.apache.org/jira/browse/HADOOP-13892 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Major > > S3 now supports object tags, attributes which can be updated during the life > of an object. > S3A could use that to preserve the permissions/ACLs of objects when copied > from elsewhere, in particular from HDFS. This would ensure that data backed > up from HDFS preserves all the permission information needed when doing a > recovery from S3 to HDFS. > Azure WASB does exactly this already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13892) use s3 tags/headers to record permissions on objects, so preserving them through distcp round trips
[
https://issues.apache.org/jira/browse/HADOOP-13892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360737#comment-16360737
]
Steve Loughran commented on HADOOP-13892:
-
Given the need for xattrs, for full distcp we really need distcp itself to
downgrade and persist the attr data as some explicitly created file alongside
the data, e.g
{code}
/path/file1.avro
/path/file1.avro.xattr
{code}
Going to close this as a wontfix
> use s3 tags/headers to record permissions on objects, so preserving them
> through distcp round trips
> ---
>
> Key: HADOOP-13892
> URL: https://issues.apache.org/jira/browse/HADOOP-13892
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Major
>
> S3 now supports object tags, attributes which can be updated during the life
> of an object.
> S3A could use that to preserve the permissions/ACLs of objects when copied
> from elsewhere, in particular from HDFS. This would ensure that data backed
> up from HDFS preserves all the permission information needed when doing a
> recovery from S3 to HDFS.
> Azure WASB does exactly this already.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13648) s3a home directory to be "/"
[
https://issues.apache.org/jira/browse/HADOOP-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-13648.
-
Resolution: Won't Fix
too late to fix this, I suspect
> s3a home directory to be "/"
>
>
> Key: HADOOP-13648
> URL: https://issues.apache.org/jira/browse/HADOOP-13648
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
>
> The home directory of an s3a instances is {{/user/" +
> System.getProperty("user.name"))}}. As HADOOP-12774 notes, it gets the user
> wrong: if it were to be correct it should use the shortname of the current
> principal.
> I don't think the username is valid here at all. s3a buckets are not
> filesystems with users and permissions; all this per-user home dir appears to
> do is cause confusion, and end up putting the output of an {{hadoop fs -rm}}
> operation into a directory under it.
> If we made it "/" then it'd be the same for all users, and "/.Trash" would be
> where deleted files get copied to
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13884) s3a create(overwrite=true) to only look for dir/ and list entries, not file
[
https://issues.apache.org/jira/browse/HADOOP-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13884:
Parent Issue: HADOOP-15220 (was: HADOOP-14831)
> s3a create(overwrite=true) to only look for dir/ and list entries, not file
> ---
>
> Key: HADOOP-13884
> URL: https://issues.apache.org/jira/browse/HADOOP-13884
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
>
> before doing a create(), s3a does a getFileStatus() to make sure there isn't
> a directory there, and, if overwrite=false, that there isn't a file.
> Because S3 caches negative HEAD/GET requests, if there isn't a file, then
> even after the PUT, a later GET/HEAD may return 404; we are generating create
> consistency where none need exist.
> when overwrite=true we don't care whether the file exists or not, only that
> the path isn't a directory. So we can just do the HEAD path +"/' and the LIST
> calls, skipping the {{HEAD path}}. This will save an HTTP round trip of a few
> hundred millis, and ensure that there's no 404 cached in the S3 front end for
> later callers
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13884) s3a create(overwrite=true) to only look for dir/ and list entries, not file
[
https://issues.apache.org/jira/browse/HADOOP-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360726#comment-16360726
]
Steve Loughran commented on HADOOP-13884:
-
* Moot for S3Guard
* For raw S3, that initial HEAD $path is faster than doing a list, so if you
are overwriting a file you are doing a HEAD $path/ and a LIST path (two calls,
one slower), and there'll be no negative caching anyway.
We'd be increasing the cost of overwriting an existing file, but reducing the
cost of creating a new file and risk of caching 404, so in some codepaths
things would get better, some would get worse
The safest way to handle this would be add a hint in file creation to say "this
create doesn't expect the dest to exist" vs "this create expects a file at the
dest" or even "don't bother with any exists checks at all", which is what is
secretly done in the commit code via WriteOperationsHelper offering a put() API
call.
> s3a create(overwrite=true) to only look for dir/ and list entries, not file
> ---
>
> Key: HADOOP-13884
> URL: https://issues.apache.org/jira/browse/HADOOP-13884
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
>
> before doing a create(), s3a does a getFileStatus() to make sure there isn't
> a directory there, and, if overwrite=false, that there isn't a file.
> Because S3 caches negative HEAD/GET requests, if there isn't a file, then
> even after the PUT, a later GET/HEAD may return 404; we are generating create
> consistency where none need exist.
> when overwrite=true we don't care whether the file exists or not, only that
> the path isn't a directory. So we can just do the HEAD path +"/' and the LIST
> calls, skipping the {{HEAD path}}. This will save an HTTP round trip of a few
> hundred millis, and ensure that there's no 404 cached in the S3 front end for
> later callers
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13371) S3A globber to use bulk listObject call over recursive directory scan
[
https://issues.apache.org/jira/browse/HADOOP-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-13371.
-
Resolution: Won't Fix
> S3A globber to use bulk listObject call over recursive directory scan
> -
>
> Key: HADOOP-13371
> URL: https://issues.apache.org/jira/browse/HADOOP-13371
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> HADOOP-13208 produces O(1) listing of directory trees in
> {{FileSystem.listStatus}} calls, but doesn't do anything for
> {{FileSystem.globStatus()}}, which uses a completely different codepath, one
> which does a selective recursive scan by pattern matching as it goes down,
> filtering out those patterns which don't match. Cost is
> O(matching-directories) + cost of examining the files.
> It should be possible to do the glob status listing in S3A not through the
> filtered treewalk, but through a list + filter operation. This would be an
> O(files) lookup *before any filtering took place*.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13371) S3A globber to use bulk listObject call over recursive directory scan
[
https://issues.apache.org/jira/browse/HADOOP-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360718#comment-16360718
]
ASF GitHub Bot commented on HADOOP-13371:
-
Github user steveloughran commented on the issue:
https://github.com/apache/hadoop/pull/204
wontfix. S3guard is needed for consistency, and as it delivers the speedup
we need at the same time, making traumatic changes to the core code is hard to
justify right now
> S3A globber to use bulk listObject call over recursive directory scan
> -
>
> Key: HADOOP-13371
> URL: https://issues.apache.org/jira/browse/HADOOP-13371
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> HADOOP-13208 produces O(1) listing of directory trees in
> {{FileSystem.listStatus}} calls, but doesn't do anything for
> {{FileSystem.globStatus()}}, which uses a completely different codepath, one
> which does a selective recursive scan by pattern matching as it goes down,
> filtering out those patterns which don't match. Cost is
> O(matching-directories) + cost of examining the files.
> It should be possible to do the glob status listing in S3A not through the
> filtered treewalk, but through a list + filter operation. This would be an
> O(files) lookup *before any filtering took place*.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13371) S3A globber to use bulk listObject call over recursive directory scan
[
https://issues.apache.org/jira/browse/HADOOP-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360716#comment-16360716
]
Steve Loughran commented on HADOOP-13371:
-
thinking of closing PR 204 and tagging this as a wontfix, at least for now.
It's not that I'm a fan of the current globber code, its that
# with HADOOP-13345 in auth mode dynamoDB offers the high performance
recursive treewalk the current globber needs
# and the consistency which FS client code demands
Given that #2 is an absolute requirement for safe use of S3 as a source of data
in any workflow which chains together queries, it's hard to justify going near
this.
sorry, thanks for your contribution here, but, like mine, it's safest to put
aside.
We do have lots of other outstanding S3 tasks (HADOOP-14831, HADOOP-15220)
which need care and attention
> S3A globber to use bulk listObject call over recursive directory scan
> -
>
> Key: HADOOP-13371
> URL: https://issues.apache.org/jira/browse/HADOOP-13371
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> HADOOP-13208 produces O(1) listing of directory trees in
> {{FileSystem.listStatus}} calls, but doesn't do anything for
> {{FileSystem.globStatus()}}, which uses a completely different codepath, one
> which does a selective recursive scan by pattern matching as it goes down,
> filtering out those patterns which don't match. Cost is
> O(matching-directories) + cost of examining the files.
> It should be possible to do the glob status listing in S3A not through the
> filtered treewalk, but through a list + filter operation. This would be an
> O(files) lookup *before any filtering took place*.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13371) S3A globber to use bulk listObject call over recursive directory scan
[
https://issues.apache.org/jira/browse/HADOOP-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360717#comment-16360717
]
ASF GitHub Bot commented on HADOOP-13371:
-
Github user steveloughran closed the pull request at:
https://github.com/apache/hadoop/pull/204
> S3A globber to use bulk listObject call over recursive directory scan
> -
>
> Key: HADOOP-13371
> URL: https://issues.apache.org/jira/browse/HADOOP-13371
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> HADOOP-13208 produces O(1) listing of directory trees in
> {{FileSystem.listStatus}} calls, but doesn't do anything for
> {{FileSystem.globStatus()}}, which uses a completely different codepath, one
> which does a selective recursive scan by pattern matching as it goes down,
> filtering out those patterns which don't match. Cost is
> O(matching-directories) + cost of examining the files.
> It should be possible to do the glob status listing in S3A not through the
> filtered treewalk, but through a list + filter operation. This would be an
> O(files) lookup *before any filtering took place*.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12020) Support AWS S3 reduced redundancy storage class
[
https://issues.apache.org/jira/browse/HADOOP-12020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360706#comment-16360706
]
Steve Loughran commented on HADOOP-12020:
-
Would welcome this in Hadoop 3.1; keep test costs down.
* copy operations in rename would need to set this too
* fault injection AWS client could generate 405 exceptions to see how they map,
and how apps would handle a translation to FNFE. What would hive & spark do, as
well as distcp.
* troubleshooting.md needs to cover the issue
Given the file is lost. FNFE is the right outcome; error text needs to include
details on reduce redundancy loss
> Support AWS S3 reduced redundancy storage class
> ---
>
> Key: HADOOP-12020
> URL: https://issues.apache.org/jira/browse/HADOOP-12020
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.0
> Environment: Hadoop on AWS
>Reporter: Yann Landrin-Schweitzer
>Priority: Major
>
> Amazon S3 uses, by default, the NORMAL_STORAGE class for s3 objects.
> This offers, according to Amazon's material, 99.% reliability.
> For many applications, however, the 99.99% reliability offered by the
> REDUCED_REDUNDANCY storage class is amply sufficient, and comes with a
> significant cost saving.
> HDFS, when using the legacy s3n protocol, or the new s3a scheme, should
> support overriding the default storage class of created s3 objects so that
> users can take advantage of this cost benefit.
> This would require minor changes of the s3n and s3a drivers, using
> a configuration property fs.s3n.storage.class to override the default storage
> when desirable.
> This override could be implemented in Jets3tNativeFileSystemStore with:
> S3Object object = new S3Object(key);
> ...
> if(storageClass!=null) object.setStorageClass(storageClass);
> It would take a more complex form in s3a, e.g. setting:
> InitiateMultipartUploadRequest initiateMPURequest =
> new InitiateMultipartUploadRequest(bucket, key, om);
> if(storageClass !=null ) {
> initiateMPURequest =
> initiateMPURequest.withStorageClass(storageClass);
> }
> and similar statements in various places.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14630) Contract Tests to verify create, mkdirs and rename under a file is forbidden
[ https://issues.apache.org/jira/browse/HADOOP-14630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14630: Priority: Major (was: Minor) > Contract Tests to verify create, mkdirs and rename under a file is forbidden > > > Key: HADOOP-14630 > URL: https://issues.apache.org/jira/browse/HADOOP-14630 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/azure, fs/s3, fs/swift >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14630-001.patch, HADOOP-14630-002.patch, > HADOOP-14630-003.patch > > > Object stores can get into trouble in ways which an FS would never, do, ways > so obvious we've never done tests for them. We know what the problems are: > test for file and dir creation directly/indirectly under other files > * mkdir(file/file) > * mkdir(file/subdir) > * dir under file/subdir/subdir > * dir/dir2/file, verify dir & dir2 exist > * dir/dir2/dir3, verify dir & dir2 exist > * rename(src, file/dest) > * rename(src, file/dir/dest) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13873) log DNS addresses on s3a init
[ https://issues.apache.org/jira/browse/HADOOP-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360705#comment-16360705 ] Steve Loughran commented on HADOOP-13873: - +list https proxy settings being picked up in httpclient > log DNS addresses on s3a init > - > > Key: HADOOP-13873 > URL: https://issues.apache.org/jira/browse/HADOOP-13873 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > HADOOP-13871 has shown that network problems can kill perf, and that it's v. > hard to track down, even if you turn up the logging in hadoop.fs.s3a and > com.amazon layers to debug. > we could maybe improve things by printing out the IPAddress of the s3 > endpoint, as that could help with the network tracing. Printing from within > hadoop shows the one given to S3a, not a different one returned by any load > balancer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14707) AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade
[
https://issues.apache.org/jira/browse/HADOOP-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14707:
Priority: Major (was: Minor)
Target Version/s: 3.1.0
> AbstractContractDistCpTest to test attr preservation with -p, verify
> blobstores downgrade
> -
>
> Key: HADOOP-14707
> URL: https://issues.apache.org/jira/browse/HADOOP-14707
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/azure, fs/s3, test, tools/distcp
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14707-001.patch, HADOOP-14707-002.patch
>
>
> It *may* be that trying to use {{distcp -p}} with S3a triggers a stack trace
> {code}
> java.lang.UnsupportedOperationException: S3AFileSystem doesn't support
> getXAttrs
> at org.apache.hadoop.fs.FileSystem.getXAttrs(FileSystem.java:2559)
> at
> org.apache.hadoop.tools.util.DistCpUtils.toCopyListingFileStatus(DistCpUtils.java:322)
>
> {code}
> Add a test to {{AbstractContractDistCpTest}} to verify that this is handled
> better. What is "handle better" here? Either ignore the option or fail with
> "don't do that" text
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14556:
Assignee: Steve Loughran
Target Version/s: 3.2.0 (was: 3.1.0)
Status: Open (was: Patch Available)
> S3A to support Delegation Tokens
>
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15220) Über-jira: S3a phase V: Hadoop 3.2 features
Steve Loughran created HADOOP-15220: --- Summary: Über-jira: S3a phase V: Hadoop 3.2 features Key: HADOOP-15220 URL: https://issues.apache.org/jira/browse/HADOOP-15220 Project: Hadoop Common Issue Type: Bug Reporter: Steve Loughran Assignee: Steve Loughran Über-jira for S3A work for Hadoop 3.2.x The items from HADOOP-14831 which didn't get into Hadoop-3.1, and anything else -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13551) hook up AwsSdkMetrics to hadoop metrics
[
https://issues.apache.org/jira/browse/HADOOP-13551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13551:
Priority: Critical (was: Minor)
Target Version/s: 3.1.0
uprating to major & targeting 3.1
Rationale: we don't see throttle events handled in the SDK itself, so when the
cause of perf problems due to excessive throttling, that's not visible to the
caller, who just sees "slow". Getting at the metrics would let us say "slow,
because..."
> hook up AwsSdkMetrics to hadoop metrics
> ---
>
> Key: HADOOP-13551
> URL: https://issues.apache.org/jira/browse/HADOOP-13551
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Sean Mackrory
>Priority: Critical
>
> There's an API in {{com.amazonaws.metrics.AwsSdkMetrics}} to give access to
> the internal metrics of the AWS libraries. We might want to get at those
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13551) hook up AwsSdkMetrics to hadoop metrics
[
https://issues.apache.org/jira/browse/HADOOP-13551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360687#comment-16360687
]
Steve Loughran commented on HADOOP-13551:
-
Sean? Started this? I'd like to get it into 3.1 if possible
> hook up AwsSdkMetrics to hadoop metrics
> ---
>
> Key: HADOOP-13551
> URL: https://issues.apache.org/jira/browse/HADOOP-13551
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Sean Mackrory
>Priority: Minor
>
> There's an API in {{com.amazonaws.metrics.AwsSdkMetrics}} to give access to
> the internal metrics of the AWS libraries. We might want to get at those
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation
[
https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15040:
Priority: Blocker (was: Major)
> Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation
> ---
>
> Key: HADOOP-15040
> URL: https://issues.apache.org/jira/browse/HADOOP-15040
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
>Priority: Blocker
> Attachments: HADOOP-15040.001.patch
>
>
> My colleagues working with Yarn log aggregation found that they were getting
> this message spammed in their logs when they used an s3a:// URI for logs
> (yarn.nodemanager.remote-app-log-dir):
> {noformat}
> getting attribute Region of com.amazonaws.management:type=AwsSdkMetrics threw
> an exception
> javax.management.RuntimeMBeanException: java.lang.NullPointerException
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
> at
>
> Caused by: java.lang.NullPointerException
> at com.amazonaws.metrics.AwsSdkMetrics.getRegion(AwsSdkMetrics.java:729)
> at com.amazonaws.metrics.MetricAdmin.getRegion(MetricAdmin.java:67)
> at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> {noformat}
> This happens even though the aws sdk cloudwatch metrics reporting was
> disabled (default), which is a bug.
> I filed a [github issue|https://github.com/aws/aws-sdk-java/issues/1375|] and
> it looks like a fix should be coming around SDK release 1.11.229 or so.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13308) S3A delete and rename may fail to preserve parent directory.
[ https://issues.apache.org/jira/browse/HADOOP-13308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360683#comment-16360683 ] Steve Loughran commented on HADOOP-13308: - I think we may just have to accept that this can happen, though I doubt s3guard will be happy; it's fsck may need to catch up with reality > S3A delete and rename may fail to preserve parent directory. > > > Key: HADOOP-13308 > URL: https://issues.apache.org/jira/browse/HADOOP-13308 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Priority: Minor > > When a file or directory is deleted or renamed in S3A, and the result of that > operation makes the parent empty, S3A must store a fake directory (a pure > metadata object) at the parent to indicate that the directory still exists. > The logic for restoring fake directories is not resilient to a process death. > This may cause a directory to vanish unexpectedly after a deletion or rename > of its last child. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14736) S3AInputStream to implement an efficient skip() call through seeking
[
https://issues.apache.org/jira/browse/HADOOP-14736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-14736.
-
Resolution: Duplicate
I should fix HADOOP-14606 instead of filing the same bug whenever I look at the
code
> S3AInputStream to implement an efficient skip() call through seeking
>
>
> Key: HADOOP-14736
> URL: https://issues.apache.org/jira/browse/HADOOP-14736
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Priority: Major
>
> {{S3AInputStream}} implements skip() naively through the byte class: Reading
> and discarding all data. Efficient on classic "sequential" reads, provided
> the forward skip is <1MB. For larger skip values or on random IO, seek()
> should be used.
> After some range checks/handling past-EOF skips to seek (EOF-1), a seek()
> should handle the skip file.
> *there are no FS contract tests for skip semantics*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14606) S3AInputStream: Handle http stream skip(n) skipping < n bytes in a forward seek
[
https://issues.apache.org/jira/browse/HADOOP-14606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14606:
Priority: Critical (was: Major)
Target Version/s: 3.1.0
> S3AInputStream: Handle http stream skip(n) skipping < n bytes in a forward
> seek
> ---
>
> Key: HADOOP-14606
> URL: https://issues.apache.org/jira/browse/HADOOP-14606
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Priority: Critical
>
> There's some hints in the InputStream docs that {{skip(n)}} may skip bytes. Codepaths only seem to do this if read() returns -1, meaning end of
> stream is reached.
> If that happens when doing a forward seek via skip, then we have got our
> numbers wrong and are in trouble. Look for a negative response, log @ ERROR
> and revert to a close/reopen seek to an absolute position.
> *I have no evidence of this acutally occurring*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13330) s3a large subdir delete to do listObjects() and delete() calls in parallel
[ https://issues.apache.org/jira/browse/HADOOP-13330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360678#comment-16360678 ] Steve Loughran commented on HADOOP-13330: - it'll be fun handling failures in this world: if one delete batch fails, you need to block for all other outstanding deletes to complete, and update s3guard with the complete set of successful deletes > s3a large subdir delete to do listObjects() and delete() calls in parallel > -- > > Key: HADOOP-13330 > URL: https://issues.apache.org/jira/browse/HADOOP-13330 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > When doing deletes of large directories/directory trees, paged list+delete > calls will be made. It would be faster if the delete and list calls were done > in parallel; the next listing batch obtained while the current batch was > being deleted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14396:
Priority: Blocker (was: Major)
> Add builder interface to FileContext
>
>
> Key: HADOOP-14396
> URL: https://issues.apache.org/jira/browse/HADOOP-14396
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Blocker
> Attachments: HADOOP-14396.00.patch
>
>
> Add builder interface for {{FileContext#create}} and {{FileContext#append}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
