[jira] [Commented] (HADOOP-13805) UGI.getCurrentUser() fails if user does not have a keytab associated
[
https://issues.apache.org/jira/browse/HADOOP-13805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871313#comment-15871313
]
Yongjun Zhang commented on HADOOP-13805:
Thanks [~tucu00]. Rebased and added an INFO message when the config is enabled,
as rev10.
> UGI.getCurrentUser() fails if user does not have a keytab associated
>
>
> Key: HADOOP-13805
> URL: https://issues.apache.org/jira/browse/HADOOP-13805
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha2
>Reporter: Alejandro Abdelnur
>Assignee: Xiao Chen
> Attachments: HADOOP-13805.006.patch, HADOOP-13805.007.patch,
> HADOOP-13805.008.patch, HADOOP-13805.009.patch, HADOOP-13805.010.patch,
> HADOOP-13805.01.patch, HADOOP-13805.02.patch, HADOOP-13805.03.patch,
> HADOOP-13805.04.patch, HADOOP-13805.05.patch
>
>
> HADOOP-13558 intention was to avoid UGI from trying to renew the TGT when the
> UGI is created from an existing Subject as in that case the keytab is not
> 'own' by UGI but by the creator of the Subject.
> In HADOOP-13558 we introduced a new private UGI constructor
> {{UserGroupInformation(Subject subject, final boolean externalKeyTab)}} and
> we use with TRUE only when doing a {{UGI.loginUserFromSubject()}}.
> The problem is, when we call {{UGI.getCurrentUser()}}, and UGI was created
> via a Subject (via the {{UGI.loginUserFromSubject()}} method), we call {{new
> UserGroupInformation(subject)}} which will delegate to
> {{UserGroupInformation(Subject subject, final boolean externalKeyTab)}} and
> that will use externalKeyTab == *FALSE*.
> Then the UGI returned by {{UGI.getCurrentUser()}} will attempt to login using
> a non-existing keytab if the TGT expired.
> This problem is experienced in {{KMSClientProvider}} when used by the HDFS
> filesystem client accessing an an encryption zone.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13805) UGI.getCurrentUser() fails if user does not have a keytab associated
[
https://issues.apache.org/jira/browse/HADOOP-13805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HADOOP-13805:
---
Attachment: HADOOP-13805.010.patch
> UGI.getCurrentUser() fails if user does not have a keytab associated
>
>
> Key: HADOOP-13805
> URL: https://issues.apache.org/jira/browse/HADOOP-13805
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha2
>Reporter: Alejandro Abdelnur
>Assignee: Xiao Chen
> Attachments: HADOOP-13805.006.patch, HADOOP-13805.007.patch,
> HADOOP-13805.008.patch, HADOOP-13805.009.patch, HADOOP-13805.010.patch,
> HADOOP-13805.01.patch, HADOOP-13805.02.patch, HADOOP-13805.03.patch,
> HADOOP-13805.04.patch, HADOOP-13805.05.patch
>
>
> HADOOP-13558 intention was to avoid UGI from trying to renew the TGT when the
> UGI is created from an existing Subject as in that case the keytab is not
> 'own' by UGI but by the creator of the Subject.
> In HADOOP-13558 we introduced a new private UGI constructor
> {{UserGroupInformation(Subject subject, final boolean externalKeyTab)}} and
> we use with TRUE only when doing a {{UGI.loginUserFromSubject()}}.
> The problem is, when we call {{UGI.getCurrentUser()}}, and UGI was created
> via a Subject (via the {{UGI.loginUserFromSubject()}} method), we call {{new
> UserGroupInformation(subject)}} which will delegate to
> {{UserGroupInformation(Subject subject, final boolean externalKeyTab)}} and
> that will use externalKeyTab == *FALSE*.
> Then the UGI returned by {{UGI.getCurrentUser()}} will attempt to login using
> a non-existing keytab if the TGT expired.
> This problem is experienced in {{KMSClientProvider}} when used by the HDFS
> filesystem client accessing an an encryption zone.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871297#comment-15871297 ] Wen Yuan Chen commented on HADOOP-9969: --- Any update on this issue? I meet the same issue on Hadoop 2.7.3 with IBM JDK 1.8 > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta, 2.5.0, 2.5.2, 2.6.0, 2.6.1, 2.8.0, 2.7.1, > 2.6.2, 2.6.3 > Environment: IBM JDK7 >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14092) Typo in hadoop-aws index.md
John Zhuge created HADOOP-14092:
---
Summary: Typo in hadoop-aws index.md
Key: HADOOP-14092
URL: https://issues.apache.org/jira/browse/HADOOP-14092
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.0.0-alpha3
Reporter: John Zhuge
Priority: Trivial
In section {{Testing against different regions}}, {{contract-tests.xml}} should
be {{contract-test-options.xml}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14091) AbstractFileSystem implementaion for 'wasbs' scheme
Varada Hemeswari created HADOOP-14091: - Summary: AbstractFileSystem implementaion for 'wasbs' scheme Key: HADOOP-14091 URL: https://issues.apache.org/jira/browse/HADOOP-14091 Project: Hadoop Common Issue Type: Task Components: fs/azure Environment: humboldt Reporter: Varada Hemeswari Currently org.apache.hadoop.fs.azure.Wasb provides AbstractFileSystem implementation for 'wasb' scheme. This task refers to providing AbstractFileSystem implementation for 'wasbs' scheme -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14049) Honour AclBit flag associated to file/folder permission for Azure datalake account
[
https://issues.apache.org/jira/browse/HADOOP-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871156#comment-15871156
]
Vishwajeet Dusane commented on HADOOP-14049:
Thanks you [~liuml07] for the review and commit.
> Honour AclBit flag associated to file/folder permission for Azure datalake
> account
> --
>
> Key: HADOOP-14049
> URL: https://issues.apache.org/jira/browse/HADOOP-14049
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Fix For: 3.0.0-alpha3
>
> Attachments: HADOOP-14049-01.patch, HADOOP-14049.02.patch
>
>
> ADLS persist AclBit information on a file/folder. Since Java SDK 2.1.4 -
> AclBit value can be retrieved using {{DirectoryEntry.aclBit}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13817) Add a finite shell command timeout to ShellBasedUnixGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871062#comment-15871062
]
ASF GitHub Bot commented on HADOOP-13817:
-
Github user QwertyManiac commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/161#discussion_r101673820
--- Diff:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ShellBasedUnixGroupsMapping.java
---
@@ -225,8 +287,16 @@ protected ShellCommandExecutor
createGroupIDExecutor(String userName) {
throw new PartialGroupNameException("failed to get group id list
for " +
"user '" + userName + "'", ece);
} catch (IOException ioe) {
-throw new PartialGroupNameException("can't execute the shell
command to"
-+ " get the list of group id for user '" + userName + "'", ioe);
+String message =
+"Can't execute the shell command to " +
+"get the list of group id for user '" + userName + "'";
+if (exec2.isTimedOut()) {
+ message +=
+ " because of the command taking longer than " +
+ "the configured timeout: " + timeout + " seconds";
+ throw new PartialGroupNameException(message);
--- End diff --
Thanks, addressed in new commit. I just felt the timeout exception may look
weird, but I've dropped the line so we can be consistent in exposing the
exception at all times.
> Add a finite shell command timeout to ShellBasedUnixGroupsMapping
> -
>
> Key: HADOOP-13817
> URL: https://issues.apache.org/jira/browse/HADOOP-13817
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.6.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Minor
>
> The ShellBasedUnixGroupsMapping run various {{id}} commands via the
> ShellCommandExecutor modules without a timeout set (its set to 0, which
> implies infinite).
> If this command hangs for a long time on the OS end due to an unresponsive
> groups backend or other reasons, it also blocks the handlers that use it on
> the NameNode (or other services that use this class). That inadvertently
> causes odd timeout troubles on the client end where its forced to retry (only
> to likely run into such hangs again with every attempt until at least one
> command returns).
> It would be helpful to have a finite command timeout after which we may give
> up on the command and return the result equivalent of no groups found.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13817) Add a finite shell command timeout to ShellBasedUnixGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871061#comment-15871061
]
ASF GitHub Bot commented on HADOOP-13817:
-
Github user QwertyManiac commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/161#discussion_r101673744
--- Diff:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestShellBasedUnixGroupsMapping.java
---
@@ -22,19 +22,32 @@
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.ReflectionUtils;
+import org.apache.hadoop.util.Shell;
import org.apache.hadoop.util.Shell.ExitCodeException;
import org.apache.hadoop.util.Shell.ShellCommandExecutor;
import org.junit.Test;
+
import static org.junit.Assert.*;
import static org.mockito.Mockito.doNothing;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
public class TestShellBasedUnixGroupsMapping {
- private static final Log LOG =
+ private static final Log TESTLOG =
LogFactory.getLog(TestShellBasedUnixGroupsMapping.class);
+ private final GenericTestUtils.LogCapturer shellMappingLog =
+ GenericTestUtils.LogCapturer.captureLogs(
+ ShellBasedUnixGroupsMapping.LOG);
+
+ private static final boolean WINDOWS =
+ (Shell.osType == Shell.OSType.OS_TYPE_WIN);
--- End diff --
Thank you, addressed in new commit.
> Add a finite shell command timeout to ShellBasedUnixGroupsMapping
> -
>
> Key: HADOOP-13817
> URL: https://issues.apache.org/jira/browse/HADOOP-13817
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.6.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Minor
>
> The ShellBasedUnixGroupsMapping run various {{id}} commands via the
> ShellCommandExecutor modules without a timeout set (its set to 0, which
> implies infinite).
> If this command hangs for a long time on the OS end due to an unresponsive
> groups backend or other reasons, it also blocks the handlers that use it on
> the NameNode (or other services that use this class). That inadvertently
> causes odd timeout troubles on the client end where its forced to retry (only
> to likely run into such hangs again with every attempt until at least one
> command returns).
> It would be helpful to have a finite command timeout after which we may give
> up on the command and return the result equivalent of no groups found.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14040) Use shaded aws-sdk uber-JAR 1.11.86
[ https://issues.apache.org/jira/browse/HADOOP-14040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871012#comment-15871012 ] Hudson commented on HADOOP-14040: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11271 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11271/]) HADOOP-14040. Use shaded aws-sdk uber-JAR 1.11.86. Contributed by Steve (liuml07: rev 658702efffdf52cf5ddf8e92f959f1157c95a348) * (edit) hadoop-project/pom.xml * (edit) hadoop-tools/hadoop-aws/pom.xml > Use shaded aws-sdk uber-JAR 1.11.86 > --- > > Key: HADOOP-14040 > URL: https://issues.apache.org/jira/browse/HADOOP-14040 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HADOOP-14040.001.patch, HADOOP-14040-branch-2-001.patch, > HADOOP-14040-branch-2.002.patch, HADOOP-14040-HADOOP-13345.001.patch > > > AWS SDK now has a (v. large) uberjar shading all dependencies > This ensures that AWS dependency changes (e.g json) don't cause problems > downstream in things like HBase, so enabling backporting if desired. > This will let us addess the org.json don't be evil problem: this SDK version > doesn't have those files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13321) Deprecate FileSystem APIs that promote inefficient call patterns.
[
https://issues.apache.org/jira/browse/HADOOP-13321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871011#comment-15871011
]
Hadoop QA commented on HADOOP-13321:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 43s{color}
| {color:red} root generated 93 new + 702 unchanged - 0 fixed = 795 total (was
702) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 37s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestKDiag |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13321 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853171/HADOOP-13321.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1cf8c4a6d9e4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f432999 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11649/artifact/patchprocess/diff-compile-javac-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11649/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11649/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11649/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Deprecate FileSystem APIs that promote inefficient call patterns.
>
[jira] [Commented] (HADOOP-14019) fix some typos in the s3a docs
[ https://issues.apache.org/jira/browse/HADOOP-14019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870985#comment-15870985 ] Hudson commented on HADOOP-14019: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11270 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11270/]) HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve (liuml07: rev bdad8b7b97d7f48119f016d68f32982d680c8796) * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md > fix some typos in the s3a docs > -- > > Key: HADOOP-14019 > URL: https://issues.apache.org/jira/browse/HADOOP-14019 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.8.0 > > Attachments: HADOOP-14019-001.patch > > > There's a few errors in the s3a docs, including one cut-and-paste error > related to the per-bucket config and JCEKS files which is potentially > misleading. > fix -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14090) Allow users to specify region for DynamoDB table instead of endpoint
[ https://issues.apache.org/jira/browse/HADOOP-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870984#comment-15870984 ] Aaron Fabbri commented on HADOOP-14090: --- Some quick comments before doing a more in-depth review: - Now is good time to clean up CLI arguments, since compatibility will be a concern post-merge. I'm fine with some of that landing in this patch. - Region and endpoint are mutually exclusive from everything I've read (e.g. [dev guide|http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-region-selection.html]). Region is short-hand for endpoint. We should specify behavior if both are specified: Log an error and ignore region (use endpoint) if both configuration parameters are set. - We should also streamline the init and config code so these things are set up in one place (e.g. the region lookup you mention). > Allow users to specify region for DynamoDB table instead of endpoint > > > Key: HADOOP-14090 > URL: https://issues.apache.org/jira/browse/HADOOP-14090 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Sean Mackrory > Attachments: HADOOP-14090-HADOOP-13345.001.patch > > > Assuming the AWS SDK allows this, I think this would be a better way to > configure it for any usage on AWS itself (with endpoint still being an option > for AWS-compatible non-AWS use cases). Unless users actually care about a > specific endpoint, this is easier. Perhaps less important, HADOOP-14023 shows > that inferring the region from the endpoint (which granted, isn't that > necessary) doesn't work very well at all. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14040) Use shaded aws-sdk uber-JAR 1.11.86
[
https://issues.apache.org/jira/browse/HADOOP-14040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14040:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha3
2.9.0
Status: Resolved (was: Patch Available)
I have committed this to {{trunk}} and {{branch-2}} branches. Thanks for your
contribution [~ste...@apache.org] and [~mackrorysd].
For S3Guard changes, we should fix in separate subtasks of the feature branch.
> Use shaded aws-sdk uber-JAR 1.11.86
> ---
>
> Key: HADOOP-14040
> URL: https://issues.apache.org/jira/browse/HADOOP-14040
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, fs/s3
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HADOOP-14040.001.patch, HADOOP-14040-branch-2-001.patch,
> HADOOP-14040-branch-2.002.patch, HADOOP-14040-HADOOP-13345.001.patch
>
>
> AWS SDK now has a (v. large) uberjar shading all dependencies
> This ensures that AWS dependency changes (e.g json) don't cause problems
> downstream in things like HBase, so enabling backporting if desired.
> This will let us addess the org.json don't be evil problem: this SDK version
> doesn't have those files.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14040) Use shaded aws-sdk uber-JAR 1.11.86
[ https://issues.apache.org/jira/browse/HADOOP-14040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870963#comment-15870963 ] Mingliang Liu commented on HADOOP-14040: +1 Will commit shortly. > Use shaded aws-sdk uber-JAR 1.11.86 > --- > > Key: HADOOP-14040 > URL: https://issues.apache.org/jira/browse/HADOOP-14040 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14040.001.patch, HADOOP-14040-branch-2-001.patch, > HADOOP-14040-branch-2.002.patch, HADOOP-14040-HADOOP-13345.001.patch > > > AWS SDK now has a (v. large) uberjar shading all dependencies > This ensures that AWS dependency changes (e.g json) don't cause problems > downstream in things like HBase, so enabling backporting if desired. > This will let us addess the org.json don't be evil problem: this SDK version > doesn't have those files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14019) fix some typos in the s3a docs
[
https://issues.apache.org/jira/browse/HADOOP-14019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14019:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I have committed this patch to {{trunk}} through {{branch-2.8}} branches.
Thanks [~ste...@apache.org] for your contribution.
> fix some typos in the s3a docs
> --
>
> Key: HADOOP-14019
> URL: https://issues.apache.org/jira/browse/HADOOP-14019
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HADOOP-14019-001.patch
>
>
> There's a few errors in the s3a docs, including one cut-and-paste error
> related to the per-bucket config and JCEKS files which is potentially
> misleading.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14019) fix some typos in the s3a docs
[ https://issues.apache.org/jira/browse/HADOOP-14019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870953#comment-15870953 ] Mingliang Liu commented on HADOOP-14019: +1 > fix some typos in the s3a docs > -- > > Key: HADOOP-14019 > URL: https://issues.apache.org/jira/browse/HADOOP-14019 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.8.0 > > Attachments: HADOOP-14019-001.patch > > > There's a few errors in the s3a docs, including one cut-and-paste error > related to the per-bucket config and JCEKS files which is potentially > misleading. > fix -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14068) Add integration test version of TestMetadataStore for DynamoDB
[
https://issues.apache.org/jira/browse/HADOOP-14068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870948#comment-15870948
]
Aaron Fabbri commented on HADOOP-14068:
---
Thanks for doing this. It is clear there is value in also running the
MetadataStore tests against the live service since it behaves differently.
{noformat}
@@ -733,12 +733,17 @@ void provisionTable(Long readCapacity, Long writeCapacity)
.withReadCapacityUnits(readCapacity)
.withWriteCapacityUnits(writeCapacity);
try {
- final ProvisionedThroughputDescription p =
- table.updateTable(toProvision).getProvisionedThroughput();
+ table.updateTable(toProvision).getProvisionedThroughput();
+ table.waitForActive();
+ final ProvisionedThroughputDescription p = table.getDescription()
+ .getProvisionedThroughput();
LOG.info("Provision table {} in region {}: readCapacityUnits={}, "
+ "writeCapacityUnits={}",
tableName, region, p.getReadCapacityUnits(),
p.getWriteCapacityUnits());
{noformat}
I assume this change is because the table moves out of the Active state after
changing throughput provisioning? So we wait for it to become Active before
proceeding? And I assume we never hit the issue with the local dynamoDB?
{noformat}
+} catch (InterruptedException e) {
+ LOG.error("Interrupted while reprovisioning I/O for table; " +
+ "may not have taken effect yet...");
{noformat}
Don't think we want to swallow this exception. How about resetting the
thread's interrupt flag then throwing an InterruptedIOException that wraps the
original?
{noformat}
conf.unset(Constants.S3GUARD_DDB_TABLE_NAME_KEY);
-conf.unset(Constants.S3GUARD_DDB_ENDPOINT_KEY);
+//conf.unset(Constants.S3GUARD_DDB_ENDPOINT_KEY);
try {
{noformat}
Just remove the line..
{noformat}
-try {
+/*try {
DynamoDBMetadataStore ddbms = new DynamoDBMetadataStore();
ddbms.initialize(conf);
fail("Should have failed because as the endpoint is not set!");
} catch (IllegalArgumentException ignored) {
-}
-// config endpoint
-conf.set(Constants.S3GUARD_DDB_ENDPOINT_KEY, ddbEndpoint);
-// config credentials
-conf.set(Constants.ACCESS_KEY, "dummy-access-key");
-conf.set(Constants.SECRET_KEY, "dummy-secret-key");
-conf.setBoolean(Constants.S3GUARD_DDB_TABLE_CREATE_KEY, true);
+}*/
+customizeConfigurationForDynamoDB(conf);
{noformat}
Ditto here.. looks like that code moved to the helper function.
> Add integration test version of TestMetadataStore for DynamoDB
> --
>
> Key: HADOOP-14068
> URL: https://issues.apache.org/jira/browse/HADOOP-14068
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14068-HADOOP-13345.001.patch,
> HADOOP-14068-HADOOP-13345.002.patch
>
>
> I tweaked TestDynamoDBMetadataStore to run against the actual Amazon DynamoDB
> service (as opposed to the "local" edition). Several tests failed because of
> minor variations in behavior. I think the differences that are clearly
> possible are enough to warrant extending that class as an ITest (but
> obviously keeping the existing test so 99% of the the coverage remains even
> when not configured for actual DynamoDB usage).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13321) Deprecate FileSystem APIs that promote inefficient call patterns.
[
https://issues.apache.org/jira/browse/HADOOP-13321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13321:
---
Affects Version/s: 3.0.0-alpha3
Status: Patch Available (was: Open)
> Deprecate FileSystem APIs that promote inefficient call patterns.
> -
>
> Key: HADOOP-13321
> URL: https://issues.apache.org/jira/browse/HADOOP-13321
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 3.0.0-alpha3
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
> Attachments: HADOOP-13321.000.patch, HADOOP-13321.001.patch,
> HADOOP-13321.002.patch
>
>
> {{FileSystem}} contains several methods that act as convenience wrappers over
> calling {{getFileStatus}} and retrieving a single property of the returned
> {{FileStatus}}. These methods have a habit of fostering inefficient call
> patterns in applications, resulting in multiple redundant {{getFileStatus}}
> calls. For HDFS, this translates into wasteful NameNode RPC traffic. For
> file systems backed by cloud object stores, this translates into wasteful
> HTTP traffic. This issue proposes to deprecate these methods and instead
> encourage applications to call {{getFileStatus}} and then reuse the same
> {{FileStatus}} instance as needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13321) Deprecate FileSystem APIs that promote inefficient call patterns.
[
https://issues.apache.org/jira/browse/HADOOP-13321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13321:
---
Attachment: HADOOP-13321.002.patch
Thanks [~ste...@apache.org] for your review. I found some of the javadoc
problems have been fixed by your previous patch. As we don't really honor the
order of javadoc annotations, the v2 patch does not change the existing
@deprecated annotation.
> Deprecate FileSystem APIs that promote inefficient call patterns.
> -
>
> Key: HADOOP-13321
> URL: https://issues.apache.org/jira/browse/HADOOP-13321
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 3.0.0-alpha3
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
> Attachments: HADOOP-13321.000.patch, HADOOP-13321.001.patch,
> HADOOP-13321.002.patch
>
>
> {{FileSystem}} contains several methods that act as convenience wrappers over
> calling {{getFileStatus}} and retrieving a single property of the returned
> {{FileStatus}}. These methods have a habit of fostering inefficient call
> patterns in applications, resulting in multiple redundant {{getFileStatus}}
> calls. For HDFS, this translates into wasteful NameNode RPC traffic. For
> file systems backed by cloud object stores, this translates into wasteful
> HTTP traffic. This issue proposes to deprecate these methods and instead
> encourage applications to call {{getFileStatus}} and then reuse the same
> {{FileStatus}} instance as needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14060) KMS /logs servlet should have access control
[
https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge reopened HADOOP-14060:
-
> KMS /logs servlet should have access control
>
>
> Key: HADOOP-14060
> URL: https://issues.apache.org/jira/browse/HADOOP-14060
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.0.0-alpha3
>Reporter: John Zhuge
>Assignee: John Zhuge
>
> HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine
> for /conf, /jmx, /logLevel, and /stacks, but not for /logs.
> The code in {{AdminAuthorizedServlet#doGet}} for /logs and
> {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that
> /logs should subject to the same access control as intended by the original
> developer.
> IMHO this could either be my misconfiguration or there is a bug somewhere in
> {{HttpServer2}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14041) CLI command to prune old metadata
[
https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870930#comment-15870930
]
Aaron Fabbri commented on HADOOP-14041:
---
This usage also doesn't work:
{noformat}
$ hadoop s3a prune -H 2 -m dynamodb://fabbri-bucket s3a://fabbri-bucket
2017-02-16 14:02:26,320 INFO s3guard.S3GuardTool: create metadata store:
dynamodb://fabbri-dev scheme: dynamodb
2017-02-16 14:02:26,456 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2017-02-16 14:02:27,191 ERROR s3guard.DynamoDBClientFactory: Incorrect DynamoDB
endpoint: null
java.lang.IllegalArgumentException: endpoint cannot be null
at com.amazonaws.util.RuntimeHttpUtils.toUri(RuntimeHttpUtils.java:147)
at
com.amazonaws.AmazonWebServiceClient.toURI(AmazonWebServiceClient.java:224)
{noformat}
Similar error doing {{hadoop s3a prune -H 2 -m dynamodb://fabbri-bucket -e
dynamodb.us-west-2.amazonaws.com}}
> CLI command to prune old metadata
> -
>
> Key: HADOOP-14041
> URL: https://issues.apache.org/jira/browse/HADOOP-14041
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14041-HADOOP-13345.001.patch,
> HADOOP-14041-HADOOP-13345.002.patch, HADOOP-14041-HADOOP-13345.003.patch,
> HADOOP-14041-HADOOP-13345.004.patch, HADOOP-14041-HADOOP-13345.005.patch
>
>
> Add a CLI command that allows users to specify an age at which to prune
> metadata that hasn't been modified for an extended period of time. Since the
> primary use-case targeted at the moment is list consistency, it would make
> sense (especially when authoritative=false) to prune metadata that is
> expected to have become consistent a long time ago.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13805) UGI.getCurrentUser() fails if user does not have a keytab associated
[
https://issues.apache.org/jira/browse/HADOOP-13805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870923#comment-15870923
]
Alejandro Abdelnur commented on HADOOP-13805:
-
I've got confirmation the patch is working as expected on a live cluster. IMO,
we can go for it. +1 again.
> UGI.getCurrentUser() fails if user does not have a keytab associated
>
>
> Key: HADOOP-13805
> URL: https://issues.apache.org/jira/browse/HADOOP-13805
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha2
>Reporter: Alejandro Abdelnur
>Assignee: Xiao Chen
> Attachments: HADOOP-13805.006.patch, HADOOP-13805.007.patch,
> HADOOP-13805.008.patch, HADOOP-13805.009.patch, HADOOP-13805.01.patch,
> HADOOP-13805.02.patch, HADOOP-13805.03.patch, HADOOP-13805.04.patch,
> HADOOP-13805.05.patch
>
>
> HADOOP-13558 intention was to avoid UGI from trying to renew the TGT when the
> UGI is created from an existing Subject as in that case the keytab is not
> 'own' by UGI but by the creator of the Subject.
> In HADOOP-13558 we introduced a new private UGI constructor
> {{UserGroupInformation(Subject subject, final boolean externalKeyTab)}} and
> we use with TRUE only when doing a {{UGI.loginUserFromSubject()}}.
> The problem is, when we call {{UGI.getCurrentUser()}}, and UGI was created
> via a Subject (via the {{UGI.loginUserFromSubject()}} method), we call {{new
> UserGroupInformation(subject)}} which will delegate to
> {{UserGroupInformation(Subject subject, final boolean externalKeyTab)}} and
> that will use externalKeyTab == *FALSE*.
> Then the UGI returned by {{UGI.getCurrentUser()}} will attempt to login using
> a non-existing keytab if the TGT expired.
> This problem is experienced in {{KMSClientProvider}} when used by the HDFS
> filesystem client accessing an an encryption zone.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14049) Honour AclBit flag associated to file/folder permission for Azure datalake account
[
https://issues.apache.org/jira/browse/HADOOP-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870908#comment-15870908
]
Hudson commented on HADOOP-14049:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11269 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11269/])
HADOOP-14049. Honour AclBit flag associated to file/folder permission (liuml07:
rev f4329990250bed62efdebe3ce2bc740092cf9573)
* (edit)
hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestListStatus.java
* (edit)
hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestADLResponseData.java
* (edit)
hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestGetFileStatus.java
* (edit)
hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java
> Honour AclBit flag associated to file/folder permission for Azure datalake
> account
> --
>
> Key: HADOOP-14049
> URL: https://issues.apache.org/jira/browse/HADOOP-14049
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Fix For: 3.0.0-alpha3
>
> Attachments: HADOOP-14049-01.patch, HADOOP-14049.02.patch
>
>
> ADLS persist AclBit information on a file/folder. Since Java SDK 2.1.4 -
> AclBit value can be retrieved using {{DirectoryEntry.aclBit}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14081) S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock)
[
https://issues.apache.org/jira/browse/HADOOP-14081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey reassigned HADOOP-14081:
-
Assignee: Rajesh Balamohan
> S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock)
> --
>
> Key: HADOOP-14081
> URL: https://issues.apache.org/jira/browse/HADOOP-14081
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HADOOP-14081.001.patch
>
>
> In {{S3ADataBlocks::ByteArrayBlock}}, data is copied whenever {{startUpload}}
> is called. It might be possible to directly access the byte[] array from
> ByteArrayOutputStream.
> Might have to extend ByteArrayOutputStream and create a method like
> getInputStream() which can return ByteArrayInputStream. This would avoid
> expensive array copy during large upload.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14046) Metastore destruction test creates table without version marker
[ https://issues.apache.org/jira/browse/HADOOP-14046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870873#comment-15870873 ] Aaron Fabbri commented on HADOOP-14046: --- +1, and committed to feature branch. Thanks for your work on this Sean. I tested this in US West 2 before committing. > Metastore destruction test creates table without version marker > --- > > Key: HADOOP-14046 > URL: https://issues.apache.org/jira/browse/HADOOP-14046 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HADOOP-14046-HADOOP-13345.001.patch > > > ITestS3GuardToolDynamoDB.testDestroyDynamoDBMetadataStore fails after > HADOOP-13985. > The gist of that test is that it creates a table directly in DynamoDB then > tries to destroy it through S3Guard CLI tools. Since there's another test > that tries to create a table through S3Guard CLI tools then deletes it > directly in DynamoDB, I propose just combing the two, so S3Guard CLI tools > are used for init and destroy, and we only use other tools to check for all > the correct side-effects. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14046) Metastore destruction test creates table without version marker
[ https://issues.apache.org/jira/browse/HADOOP-14046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14046: -- Release Note: (was: +1, and committed to feature branch. Thanks for your work on this Sean. I tested this in US West 2 before committing.) > Metastore destruction test creates table without version marker > --- > > Key: HADOOP-14046 > URL: https://issues.apache.org/jira/browse/HADOOP-14046 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HADOOP-14046-HADOOP-13345.001.patch > > > ITestS3GuardToolDynamoDB.testDestroyDynamoDBMetadataStore fails after > HADOOP-13985. > The gist of that test is that it creates a table directly in DynamoDB then > tries to destroy it through S3Guard CLI tools. Since there's another test > that tries to create a table through S3Guard CLI tools then deletes it > directly in DynamoDB, I propose just combing the two, so S3Guard CLI tools > are used for init and destroy, and we only use other tools to check for all > the correct side-effects. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14046) Metastore destruction test creates table without version marker
[ https://issues.apache.org/jira/browse/HADOOP-14046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14046: -- Resolution: Fixed Release Note: +1, and committed to feature branch. Thanks for your work on this Sean. I tested this in US West 2 before committing. Status: Resolved (was: Patch Available) > Metastore destruction test creates table without version marker > --- > > Key: HADOOP-14046 > URL: https://issues.apache.org/jira/browse/HADOOP-14046 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HADOOP-14046-HADOOP-13345.001.patch > > > ITestS3GuardToolDynamoDB.testDestroyDynamoDBMetadataStore fails after > HADOOP-13985. > The gist of that test is that it creates a table directly in DynamoDB then > tries to destroy it through S3Guard CLI tools. Since there's another test > that tries to create a table through S3Guard CLI tools then deletes it > directly in DynamoDB, I propose just combing the two, so S3Guard CLI tools > are used for init and destroy, and we only use other tools to check for all > the correct side-effects. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14049) Honour AclBit flag associated to file/folder permission for Azure datalake account
[
https://issues.apache.org/jira/browse/HADOOP-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14049:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha3
Status: Resolved (was: Patch Available)
I have committed the patch to {{trunk}} branch. Thanks for your contribution
[~vishwajeet.dusane].
> Honour AclBit flag associated to file/folder permission for Azure datalake
> account
> --
>
> Key: HADOOP-14049
> URL: https://issues.apache.org/jira/browse/HADOOP-14049
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Fix For: 3.0.0-alpha3
>
> Attachments: HADOOP-14049-01.patch, HADOOP-14049.02.patch
>
>
> ADLS persist AclBit information on a file/folder. Since Java SDK 2.1.4 -
> AclBit value can be retrieved using {{DirectoryEntry.aclBit}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14049) Honour AclBit flag associated to file/folder permission for Azure datalake account
[
https://issues.apache.org/jira/browse/HADOOP-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870846#comment-15870846
]
Mingliang Liu commented on HADOOP-14049:
+1
> Honour AclBit flag associated to file/folder permission for Azure datalake
> account
> --
>
> Key: HADOOP-14049
> URL: https://issues.apache.org/jira/browse/HADOOP-14049
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: HADOOP-14049-01.patch, HADOOP-14049.02.patch
>
>
> ADLS persist AclBit information on a file/folder. Since Java SDK 2.1.4 -
> AclBit value can be retrieved using {{DirectoryEntry.aclBit}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14090) Allow users to specify region for DynamoDB table instead of endpoint
[ https://issues.apache.org/jira/browse/HADOOP-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HADOOP-14090: --- Attachment: HADOOP-14090-HADOOP-13345.001.patch Attaching a first stab that I think basically works. Most existing tests pass and I think the only failures are unrelated. I still need to look into a few assumptions (like if there's ever a reason why you would / could specify both the region and an endpoint) and do more testing (not sure if region is something that can be tested well outside of just running tests with various values for that property in the test config - I suspect not). For the CLI tool, to specify a region instead of an endpoint (-e), -r is already used for specifying read throughput on the init command. So I renamed everything to full words. Single letters is just going to get crazier. I'm about to do some more work on the prune command where I'm considering bigger changes to the way all these options get specified, so we'll see. The other thing is that the S3 bucket's region was getting looked up in a couple of different places, so similarly this config is getting looked up in a few different layers. Might be good to reconsider that structure, but perhaps that's best done as a separate issue > Allow users to specify region for DynamoDB table instead of endpoint > > > Key: HADOOP-14090 > URL: https://issues.apache.org/jira/browse/HADOOP-14090 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Sean Mackrory > Attachments: HADOOP-14090-HADOOP-13345.001.patch > > > Assuming the AWS SDK allows this, I think this would be a better way to > configure it for any usage on AWS itself (with endpoint still being an option > for AWS-compatible non-AWS use cases). Unless users actually care about a > specific endpoint, this is easier. Perhaps less important, HADOOP-14023 shows > that inferring the region from the endpoint (which granted, isn't that > necessary) doesn't work very well at all. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14041) CLI command to prune old metadata
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870761#comment-15870761 ] Aaron Fabbri commented on HADOOP-14041: --- [~mackrorysd] I'm also fine with doing a followup "S3guard CLI improvements" JIRA.. there are multiple related issues I'd like to tackle.. So I'm fine with committing this patch (once I finish my testing) and then filing new JIRA. > CLI command to prune old metadata > - > > Key: HADOOP-14041 > URL: https://issues.apache.org/jira/browse/HADOOP-14041 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HADOOP-14041-HADOOP-13345.001.patch, > HADOOP-14041-HADOOP-13345.002.patch, HADOOP-14041-HADOOP-13345.003.patch, > HADOOP-14041-HADOOP-13345.004.patch, HADOOP-14041-HADOOP-13345.005.patch > > > Add a CLI command that allows users to specify an age at which to prune > metadata that hasn't been modified for an extended period of time. Since the > primary use-case targeted at the moment is list consistency, it would make > sense (especially when authoritative=false) to prune metadata that is > expected to have become consistent a long time ago. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14041) CLI command to prune old metadata
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870710#comment-15870710 ] Sean Mackrory commented on HADOOP-14041: Test failure #5 is addressed by HADOOP-14046. The version marker patch went in after I had last run that tests when cleaning up the original S3GuardTool tests and it breaks. I'll make the change required to have an S3 path provide that instead. I'm really surprised your last example behaves differently. I don't like the logic for determining how the CLI tools connect to the metastore and it's been an issue a couple of times now. What if we drop the "-m dynamodb:// " notion entirely and just use configuration + optional S3 path to connect? > CLI command to prune old metadata > - > > Key: HADOOP-14041 > URL: https://issues.apache.org/jira/browse/HADOOP-14041 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HADOOP-14041-HADOOP-13345.001.patch, > HADOOP-14041-HADOOP-13345.002.patch, HADOOP-14041-HADOOP-13345.003.patch, > HADOOP-14041-HADOOP-13345.004.patch, HADOOP-14041-HADOOP-13345.005.patch > > > Add a CLI command that allows users to specify an age at which to prune > metadata that hasn't been modified for an extended period of time. Since the > primary use-case targeted at the moment is list consistency, it would make > sense (especially when authoritative=false) to prune metadata that is > expected to have become consistent a long time ago. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14041) CLI command to prune old metadata
[
https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870704#comment-15870704
]
Aaron Fabbri commented on HADOOP-14041:
---
Also, shouldn't this work?
{noformat}
$ hadoop s3a -Dfs.s3a.s3guard.ddb.table=fabbri-bucket prune -H 1
s3a://fabbri-bucket
Usage: hadoop s3a [init|destroy|import|diff|prune] [OPTIONS] [ARGUMENTS]
perform metadata store administrative commands for s3a filesystem.
{noformat}
> CLI command to prune old metadata
> -
>
> Key: HADOOP-14041
> URL: https://issues.apache.org/jira/browse/HADOOP-14041
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14041-HADOOP-13345.001.patch,
> HADOOP-14041-HADOOP-13345.002.patch, HADOOP-14041-HADOOP-13345.003.patch,
> HADOOP-14041-HADOOP-13345.004.patch, HADOOP-14041-HADOOP-13345.005.patch
>
>
> Add a CLI command that allows users to specify an age at which to prune
> metadata that hasn't been modified for an extended period of time. Since the
> primary use-case targeted at the moment is list consistency, it would make
> sense (especially when authoritative=false) to prune metadata that is
> expected to have become consistent a long time ago.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14041) CLI command to prune old metadata
[
https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870696#comment-15870696
]
Aaron Fabbri commented on HADOOP-14041:
---
Just recording results from my test runs last night:
mvn clean verify -Ds3guard -Ddynamo -Dscale
Tests run: 366, Failures: 3, Errors: 2, Skipped: 70
{noformat}
Failed tests:
(1)
ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRecursiveRootListing:222->Assert.assertTrue:41->Assert.fail:88
files mismatch: "s3a://fabbri-dev/user/fabbri/test/file"
"s3a://fabbri-dev/user/fabbri/test/parentdir/child"
(2)
ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive:95->Assert.fail:88
After 1 attempts: listing after rm /* not empty
final [00] S3AFileStatus{path=s3a://fabbri-dev/Users; isDirectory=true;
modification_time=0; access_time=0; owner=fabbri; group=fabbri;
permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=false
(3)
ITestS3AContractRootDir.testListEmptyRootDirectory:63->AbstractContractRootDirectoryTest.testListEmptyRootDirectory:186->Assert.fail:88
Deleted file: unexpectedly found s3a://fabbri-dev/user as
S3AFileStatus{path=s3a://fabbri-dev/user; isDirectory=true;
modification_time=0; access_time=0; owner=fabbri; group=fabbri;
permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=false
Tests in error:
(4) ITestS3ACredentialsInURL.testInstantiateFromURL:86 » InterruptedIO
initTable: ...
(5) ITestS3GuardToolDynamoDB.testDestroyDynamoDBMetadataStore:145 » IO S3Guard
tab...
{noformat}
1-3 are root directory test failures which have been flaky.. one is leftover
files from FileSystemContractBaseTest, the other two are something creating a
user/ directory while test is running?
4 is expected: s3guard will not use URI credentials. (We should skip this if
we don't already do that in pending patch)
5 is this: S3Guard table lacks version marker. Table:
destroyDynamoDBMetadataStore-1546206104
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.verifyVersionCompatibility(DynamoDBMetadataStore.java:667)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initTable(DynamoDBMetadataStore.java:630)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:288)
I don't think any of these are related, except maybe the last one?
As for testing the prune command itself, the first thing I notice is that it
behaves a bit differently than, say, diff. Diff appears to use bucket name as
table name if one is not set, but prune requires setting the table name.
{noformat}
$ hadoop s3a prune -H 1 s3a://fabbri-bucket
No DynamoDB table name configured!
{noformat}
> CLI command to prune old metadata
> -
>
> Key: HADOOP-14041
> URL: https://issues.apache.org/jira/browse/HADOOP-14041
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14041-HADOOP-13345.001.patch,
> HADOOP-14041-HADOOP-13345.002.patch, HADOOP-14041-HADOOP-13345.003.patch,
> HADOOP-14041-HADOOP-13345.004.patch, HADOOP-14041-HADOOP-13345.005.patch
>
>
> Add a CLI command that allows users to specify an age at which to prune
> metadata that hasn't been modified for an extended period of time. Since the
> primary use-case targeted at the moment is list consistency, it would make
> sense (especially when authoritative=false) to prune metadata that is
> expected to have become consistent a long time ago.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13282) S3 blob etags to be made visible in status/getFileChecksum() calls
[
https://issues.apache.org/jira/browse/HADOOP-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870601#comment-15870601
]
Thomas Demoor commented on HADOOP-13282:
ETag:
* singlepart= md5sum
* multipart = (md5sum of the concatenation of the md5sum of each part) + "-" +
number of parts
> S3 blob etags to be made visible in status/getFileChecksum() calls
> --
>
> Key: HADOOP-13282
> URL: https://issues.apache.org/jira/browse/HADOOP-13282
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
>
> If the etags of blobs were exported via {{getFileChecksum()}}, it'd be
> possible to probe for a blob being in sync with a local file. Distcp could
> use this to decide whether to skip a file or not.
> Now, there's a problem there: distcp needs source and dest filesystems to
> implement the same algorithm. It'd only work out the box if you were copying
> between S3 instances. There are also quirks with encryption and multipart:
> [s3
> docs|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html].
> At the very least, it's something which could be used when indexing the FS,
> to check for changes later.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14028) S3A block output streams don't delete temporary files in multipart uploads
[ https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14028: Status: Patch Available (was: Open) > S3A block output streams don't delete temporary files in multipart uploads > -- > > Key: HADOOP-14028 > URL: https://issues.apache.org/jira/browse/HADOOP-14028 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: JDK 8 + ORC 1.3.0 + hadoop-aws 3.0.0-alpha2 >Reporter: Seth Fitzsimmons >Assignee: Steve Loughran >Priority: Critical > Attachments: HADOOP-14028-006.patch, HADOOP-14028-branch-2-001.patch, > HADOOP-14028-branch-2.8-002.patch, HADOOP-14028-branch-2.8-003.patch, > HADOOP-14028-branch-2.8-004.patch, HADOOP-14028-branch-2.8-005.patch > > > I have `fs.s3a.fast.upload` enabled with 3.0.0-alpha2 (it's exactly what I > was looking for after running into the same OOM problems) and don't see it > cleaning up the disk-cached blocks. > I'm generating a ~50GB file on an instance with ~6GB free when the process > starts. My expectation is that local copies of the blocks would be deleted > after those parts finish uploading, but I'm seeing more than 15 blocks in > /tmp (and none of them have been deleted thus far). > I see that DiskBlock deletes temporary files when closed, but is it closed > after individual blocks have finished uploading or when the entire file has > been fully written to the FS (full upload completed, including all parts)? > As a temporary workaround to avoid running out of space, I'm listing files, > sorting by atime, and deleting anything older than the first 20: `ls -ut | > tail -n +21 | xargs rm` > Steve Loughran says: > > They should be deleted as soon as the upload completes; the close() call > > that the AWS httpclient makes on the input stream triggers the deletion. > > Though there aren't tests for it, as I recall. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14028) S3A block output streams don't delete temporary files in multipart uploads
[ https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870560#comment-15870560 ] Steve Loughran edited comment on HADOOP-14028 at 2/16/17 7:27 PM: -- patch 006 pass file down to the put request, simplify blockUploadData and tune tests, I've been into the AWS SDK now, as well as testing what's happening in the java.io code # we MUST pass in the File instance for reliable uploads of file data. # cleanup must therefore always be in the block close() call. tested: s3 ireland, with scale tests at 128M was (Author: ste...@apache.org): patch 006 pass file down to the put request, simplify blockUploadData and tune tests, I've been into the AWS SDK now, as well as testing what's happening in the java.io code # we MUST pass in the File instance for reliable uploads of file data. # cleanup must therefore always be in the block close() call. tested: s3 ireland, with scale tests > S3A block output streams don't delete temporary files in multipart uploads > -- > > Key: HADOOP-14028 > URL: https://issues.apache.org/jira/browse/HADOOP-14028 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: JDK 8 + ORC 1.3.0 + hadoop-aws 3.0.0-alpha2 >Reporter: Seth Fitzsimmons >Assignee: Steve Loughran >Priority: Critical > Attachments: HADOOP-14028-006.patch, HADOOP-14028-branch-2-001.patch, > HADOOP-14028-branch-2.8-002.patch, HADOOP-14028-branch-2.8-003.patch, > HADOOP-14028-branch-2.8-004.patch, HADOOP-14028-branch-2.8-005.patch > > > I have `fs.s3a.fast.upload` enabled with 3.0.0-alpha2 (it's exactly what I > was looking for after running into the same OOM problems) and don't see it > cleaning up the disk-cached blocks. > I'm generating a ~50GB file on an instance with ~6GB free when the process > starts. My expectation is that local copies of the blocks would be deleted > after those parts finish uploading, but I'm seeing more than 15 blocks in > /tmp (and none of them have been deleted thus far). > I see that DiskBlock deletes temporary files when closed, but is it closed > after individual blocks have finished uploading or when the entire file has > been fully written to the FS (full upload completed, including all parts)? > As a temporary workaround to avoid running out of space, I'm listing files, > sorting by atime, and deleting anything older than the first 20: `ls -ut | > tail -n +21 | xargs rm` > Steve Loughran says: > > They should be deleted as soon as the upload completes; the close() call > > that the AWS httpclient makes on the input stream triggers the deletion. > > Though there aren't tests for it, as I recall. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14028) S3A block output streams don't delete temporary files in multipart uploads
[ https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14028: Attachment: HADOOP-14028-006.patch patch 006 pass file down to the put request, simplify blockUploadData and tune tests, I've been into the AWS SDK now, as well as testing what's happening in the java.io code # we MUST pass in the File instance for reliable uploads of file data. # cleanup must therefore always be in the block close() call. tested: s3 ireland, with scale tests > S3A block output streams don't delete temporary files in multipart uploads > -- > > Key: HADOOP-14028 > URL: https://issues.apache.org/jira/browse/HADOOP-14028 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: JDK 8 + ORC 1.3.0 + hadoop-aws 3.0.0-alpha2 >Reporter: Seth Fitzsimmons >Assignee: Steve Loughran >Priority: Critical > Attachments: HADOOP-14028-006.patch, HADOOP-14028-branch-2-001.patch, > HADOOP-14028-branch-2.8-002.patch, HADOOP-14028-branch-2.8-003.patch, > HADOOP-14028-branch-2.8-004.patch, HADOOP-14028-branch-2.8-005.patch > > > I have `fs.s3a.fast.upload` enabled with 3.0.0-alpha2 (it's exactly what I > was looking for after running into the same OOM problems) and don't see it > cleaning up the disk-cached blocks. > I'm generating a ~50GB file on an instance with ~6GB free when the process > starts. My expectation is that local copies of the blocks would be deleted > after those parts finish uploading, but I'm seeing more than 15 blocks in > /tmp (and none of them have been deleted thus far). > I see that DiskBlock deletes temporary files when closed, but is it closed > after individual blocks have finished uploading or when the entire file has > been fully written to the FS (full upload completed, including all parts)? > As a temporary workaround to avoid running out of space, I'm listing files, > sorting by atime, and deleting anything older than the first 20: `ls -ut | > tail -n +21 | xargs rm` > Steve Loughran says: > > They should be deleted as soon as the upload completes; the close() call > > that the AWS httpclient makes on the input stream triggers the deletion. > > Though there aren't tests for it, as I recall. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14028) S3A block output streams don't delete temporary files in multipart uploads
[ https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14028: Status: Open (was: Patch Available) > S3A block output streams don't delete temporary files in multipart uploads > -- > > Key: HADOOP-14028 > URL: https://issues.apache.org/jira/browse/HADOOP-14028 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: JDK 8 + ORC 1.3.0 + hadoop-aws 3.0.0-alpha2 >Reporter: Seth Fitzsimmons >Assignee: Steve Loughran >Priority: Critical > Attachments: HADOOP-14028-006.patch, HADOOP-14028-branch-2-001.patch, > HADOOP-14028-branch-2.8-002.patch, HADOOP-14028-branch-2.8-003.patch, > HADOOP-14028-branch-2.8-004.patch, HADOOP-14028-branch-2.8-005.patch > > > I have `fs.s3a.fast.upload` enabled with 3.0.0-alpha2 (it's exactly what I > was looking for after running into the same OOM problems) and don't see it > cleaning up the disk-cached blocks. > I'm generating a ~50GB file on an instance with ~6GB free when the process > starts. My expectation is that local copies of the blocks would be deleted > after those parts finish uploading, but I'm seeing more than 15 blocks in > /tmp (and none of them have been deleted thus far). > I see that DiskBlock deletes temporary files when closed, but is it closed > after individual blocks have finished uploading or when the entire file has > been fully written to the FS (full upload completed, including all parts)? > As a temporary workaround to avoid running out of space, I'm listing files, > sorting by atime, and deleting anything older than the first 20: `ls -ut | > tail -n +21 | xargs rm` > Steve Loughran says: > > They should be deleted as soon as the upload completes; the close() call > > that the AWS httpclient makes on the input stream triggers the deletion. > > Though there aren't tests for it, as I recall. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14028) S3A block output streams don't delete temporary files in multipart uploads
[
https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870545#comment-15870545
]
Steve Loughran commented on HADOOP-14028:
-
and that new test shows..file streams don't support mark/reset
{code}
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.235 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
testMarkReset(org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk) Time elapsed:
0.533 sec <<< FAILURE!
java.lang.AssertionError: Mark not supported in java.io.FileInputStream@7143d536
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray.markAndResetDatablock(ITestS3ABlockOutputArray.java:134)
at
org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray.testMarkReset(ITestS3ABlockOutputArray.java:147)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
Looks like a valid complaint.
Looking at the AWS code, they have a special class,
{{com.amazonaws.internal.ResettableInputStream}} which does the mark/reset
operation by using the FileChannel behind a File; we need to always pass it
down for mark/reset to work with file sources
> S3A block output streams don't delete temporary files in multipart uploads
> --
>
> Key: HADOOP-14028
> URL: https://issues.apache.org/jira/browse/HADOOP-14028
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: JDK 8 + ORC 1.3.0 + hadoop-aws 3.0.0-alpha2
>Reporter: Seth Fitzsimmons
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-14028-branch-2-001.patch,
> HADOOP-14028-branch-2.8-002.patch, HADOOP-14028-branch-2.8-003.patch,
> HADOOP-14028-branch-2.8-004.patch, HADOOP-14028-branch-2.8-005.patch
>
>
> I have `fs.s3a.fast.upload` enabled with 3.0.0-alpha2 (it's exactly what I
> was looking for after running into the same OOM problems) and don't see it
> cleaning up the disk-cached blocks.
> I'm generating a ~50GB file on an instance with ~6GB free when the process
> starts. My expectation is that local copies of the blocks would be deleted
> after those parts finish uploading, but I'm seeing more than 15 blocks in
> /tmp (and none of them have been deleted thus far).
> I see that DiskBlock deletes temporary files when closed, but is it closed
> after individual blocks have finished uploading or when the entire file has
> been fully written to the FS (full upload completed, including all parts)?
> As a temporary workaround to avoid running out of space, I'm listing files,
> sorting by atime, and deleting anything older than the first 20: `ls -ut |
> tail -n +21 | xargs rm`
> Steve Loughran says:
> > They should be deleted as soon as the upload completes; the close() call
> > that the AWS httpclient makes on the input stream triggers the deletion.
> > Though there aren't tests for it, as I recall.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14083) KMS should support old SSL clients
[
https://issues.apache.org/jira/browse/HADOOP-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870480#comment-15870480
]
Robert Kanter commented on HADOOP-14083:
{quote}Good news is, trunk has gotten rid of tomcat and is on jetty now!{quote}
Oh, right. I knew that.
> KMS should support old SSL clients
> --
>
> Key: HADOOP-14083
> URL: https://issues.apache.org/jira/browse/HADOOP-14083
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.8.0, 2.7.4, 2.6.6
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-14083.branch-2.001.patch
>
>
> HADOOP-13812 upgraded Tomcat to 6.0.48 which filters weak ciphers. Old SSL
> clients such as curl stop working. The symptom is {{NSS error -12286}} when
> running {{curl -v}}.
> Instead of forcing the SSL clients to upgrade, we can configure Tomcat to
> explicitly allow enough weak ciphers so that old SSL clients can work.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14090) Allow users to specify region for DynamoDB table instead of endpoint
Sean Mackrory created HADOOP-14090: -- Summary: Allow users to specify region for DynamoDB table instead of endpoint Key: HADOOP-14090 URL: https://issues.apache.org/jira/browse/HADOOP-14090 Project: Hadoop Common Issue Type: Sub-task Reporter: Sean Mackrory Assuming the AWS SDK allows this, I think this would be a better way to configure it for any usage on AWS itself (with endpoint still being an option for AWS-compatible non-AWS use cases). Unless users actually care about a specific endpoint, this is easier. Perhaps less important, HADOOP-14023 shows that inferring the region from the endpoint (which granted, isn't that necessary) doesn't work very well at all. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14089) Shaded Hadoop client runtime includes non-shaded classes
[
https://issues.apache.org/jira/browse/HADOOP-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-14089:
-
Target Version/s: 3.0.0-beta1
Priority: Critical (was: Major)
> Shaded Hadoop client runtime includes non-shaded classes
>
>
> Key: HADOOP-14089
> URL: https://issues.apache.org/jira/browse/HADOOP-14089
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha2
>Reporter: David Phillips
>Priority: Critical
>
> The jar includes things like {{assets}}, {{okio}}, {{javax/annotation}},
> {{javax/ws}}, {{mozilla}}, etc.
> An easy way to verify this is to look at the contents of the jar:
> {code}
> jar tf hadoop-client-runtime-xxx.jar | sort | grep -v '^org/apache/hadoop'
> {code}
> For standard dependencies, such as the JSR 305 {{javax.annotation}} or JAX-RS
> {{javax.ws}}, it makes sense for those to be normal dependencies in the POM
> -- they are standard, so version conflicts shouldn't be a problem. The JSR
> 305 annotations can be {{true}} since they aren't needed
> at runtime (this is what Guava does).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14083) KMS should support old SSL clients
[
https://issues.apache.org/jira/browse/HADOOP-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870473#comment-15870473
]
Allen Wittenauer commented on HADOOP-14083:
---
bq. How can I fix the shellcheck errors for the multi-line string?
It's really shellcheck giving a hint that this is doing something it shouldn't.
There's two key problems with this approach:
1) any space in that string will cause a new option to be formed on the command
line
2) the command line is going to be REALLY long and will likely blow CLI buffers
on some operating systems
Maybe this should just be a change to catalina.properties?
> KMS should support old SSL clients
> --
>
> Key: HADOOP-14083
> URL: https://issues.apache.org/jira/browse/HADOOP-14083
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.8.0, 2.7.4, 2.6.6
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-14083.branch-2.001.patch
>
>
> HADOOP-13812 upgraded Tomcat to 6.0.48 which filters weak ciphers. Old SSL
> clients such as curl stop working. The symptom is {{NSS error -12286}} when
> running {{curl -v}}.
> Instead of forcing the SSL clients to upgrade, we can configure Tomcat to
> explicitly allow enough weak ciphers so that old SSL clients can work.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13817) Add a finite shell command timeout to ShellBasedUnixGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870471#comment-15870471
]
Wei-Chiu Chuang commented on HADOOP-13817:
--
Okay I did a quick review and there were only two nits after Xiaoyu's numerous
reviews.
By the way, Yetus does not support precommit check if the patch is a github
pull request, right? I am not sure how to make that happen... Maybe attach a
patch into this jira would trigger that.
> Add a finite shell command timeout to ShellBasedUnixGroupsMapping
> -
>
> Key: HADOOP-13817
> URL: https://issues.apache.org/jira/browse/HADOOP-13817
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.6.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Minor
>
> The ShellBasedUnixGroupsMapping run various {{id}} commands via the
> ShellCommandExecutor modules without a timeout set (its set to 0, which
> implies infinite).
> If this command hangs for a long time on the OS end due to an unresponsive
> groups backend or other reasons, it also blocks the handlers that use it on
> the NameNode (or other services that use this class). That inadvertently
> causes odd timeout troubles on the client end where its forced to retry (only
> to likely run into such hangs again with every attempt until at least one
> command returns).
> It would be helpful to have a finite command timeout after which we may give
> up on the command and return the result equivalent of no groups found.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13817) Add a finite shell command timeout to ShellBasedUnixGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870468#comment-15870468
]
ASF GitHub Bot commented on HADOOP-13817:
-
Github user jojochuang commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/161#discussion_r101592507
--- Diff:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ShellBasedUnixGroupsMapping.java
---
@@ -225,8 +287,16 @@ protected ShellCommandExecutor
createGroupIDExecutor(String userName) {
throw new PartialGroupNameException("failed to get group id list
for " +
"user '" + userName + "'", ece);
} catch (IOException ioe) {
-throw new PartialGroupNameException("can't execute the shell
command to"
-+ " get the list of group id for user '" + userName + "'", ioe);
+String message =
+"Can't execute the shell command to " +
+"get the list of group id for user '" + userName + "'";
+if (exec2.isTimedOut()) {
+ message +=
+ " because of the command taking longer than " +
+ "the configured timeout: " + timeout + " seconds";
+ throw new PartialGroupNameException(message);
--- End diff --
Maybe this line is not needed if it will throw the exception anyway? The
only difference is that it will not carry the original exception
> Add a finite shell command timeout to ShellBasedUnixGroupsMapping
> -
>
> Key: HADOOP-13817
> URL: https://issues.apache.org/jira/browse/HADOOP-13817
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.6.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Minor
>
> The ShellBasedUnixGroupsMapping run various {{id}} commands via the
> ShellCommandExecutor modules without a timeout set (its set to 0, which
> implies infinite).
> If this command hangs for a long time on the OS end due to an unresponsive
> groups backend or other reasons, it also blocks the handlers that use it on
> the NameNode (or other services that use this class). That inadvertently
> causes odd timeout troubles on the client end where its forced to retry (only
> to likely run into such hangs again with every attempt until at least one
> command returns).
> It would be helpful to have a finite command timeout after which we may give
> up on the command and return the result equivalent of no groups found.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14089) Shaded Hadoop client runtime includes non-shaded classes
David Phillips created HADOOP-14089:
---
Summary: Shaded Hadoop client runtime includes non-shaded classes
Key: HADOOP-14089
URL: https://issues.apache.org/jira/browse/HADOOP-14089
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0-alpha2
Reporter: David Phillips
The jar includes things like {{assets}}, {{okio}}, {{javax/annotation}},
{{javax/ws}}, {{mozilla}}, etc.
An easy way to verify this is to look at the contents of the jar:
{code}
jar tf hadoop-client-runtime-xxx.jar | sort | grep -v '^org/apache/hadoop'
{code}
For standard dependencies, such as the JSR 305 {{javax.annotation}} or JAX-RS
{{javax.ws}}, it makes sense for those to be normal dependencies in the POM --
they are standard, so version conflicts shouldn't be a problem. The JSR 305
annotations can be {{true}} since they aren't needed at
runtime (this is what Guava does).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13817) Add a finite shell command timeout to ShellBasedUnixGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870464#comment-15870464
]
ASF GitHub Bot commented on HADOOP-13817:
-
Github user jojochuang commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/161#discussion_r101592163
--- Diff:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestShellBasedUnixGroupsMapping.java
---
@@ -22,19 +22,32 @@
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.ReflectionUtils;
+import org.apache.hadoop.util.Shell;
import org.apache.hadoop.util.Shell.ExitCodeException;
import org.apache.hadoop.util.Shell.ShellCommandExecutor;
import org.junit.Test;
+
import static org.junit.Assert.*;
import static org.mockito.Mockito.doNothing;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
public class TestShellBasedUnixGroupsMapping {
- private static final Log LOG =
+ private static final Log TESTLOG =
LogFactory.getLog(TestShellBasedUnixGroupsMapping.class);
+ private final GenericTestUtils.LogCapturer shellMappingLog =
+ GenericTestUtils.LogCapturer.captureLogs(
+ ShellBasedUnixGroupsMapping.LOG);
+
+ private static final boolean WINDOWS =
+ (Shell.osType == Shell.OSType.OS_TYPE_WIN);
--- End diff --
I think you can just use Shell.WINDOWS instead.
> Add a finite shell command timeout to ShellBasedUnixGroupsMapping
> -
>
> Key: HADOOP-13817
> URL: https://issues.apache.org/jira/browse/HADOOP-13817
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.6.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Minor
>
> The ShellBasedUnixGroupsMapping run various {{id}} commands via the
> ShellCommandExecutor modules without a timeout set (its set to 0, which
> implies infinite).
> If this command hangs for a long time on the OS end due to an unresponsive
> groups backend or other reasons, it also blocks the handlers that use it on
> the NameNode (or other services that use this class). That inadvertently
> causes odd timeout troubles on the client end where its forced to retry (only
> to likely run into such hangs again with every attempt until at least one
> command returns).
> It would be helpful to have a finite command timeout after which we may give
> up on the command and return the result equivalent of no groups found.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14083) KMS should support old SSL clients
[
https://issues.apache.org/jira/browse/HADOOP-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870459#comment-15870459
]
Xiao Chen commented on HADOOP-14083:
Good news is, trunk has gotten rid of tomcat and is on jetty now! :)
> KMS should support old SSL clients
> --
>
> Key: HADOOP-14083
> URL: https://issues.apache.org/jira/browse/HADOOP-14083
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.8.0, 2.7.4, 2.6.6
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-14083.branch-2.001.patch
>
>
> HADOOP-13812 upgraded Tomcat to 6.0.48 which filters weak ciphers. Old SSL
> clients such as curl stop working. The symptom is {{NSS error -12286}} when
> running {{curl -v}}.
> Instead of forcing the SSL clients to upgrade, we can configure Tomcat to
> explicitly allow enough weak ciphers so that old SSL clients can work.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14040) Use shaded aws-sdk uber-JAR 1.11.86
[
https://issues.apache.org/jira/browse/HADOOP-14040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870442#comment-15870442
]
Hadoop QA commented on HADOOP-14040:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
46s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
40s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
55s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 55s{color}
| {color:red} root-jdk1.8.0_121 with JDK v1.8.0_121 generated 5 new + 911
unchanged - 0 fixed = 916 total (was 911) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
46s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 46s{color}
| {color:red} root-jdk1.7.0_121 with JDK v1.7.0_121 generated 5 new + 1007
unchanged - 0 fixed = 1012 total (was 1007) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
9s{color} | {color:green} hadoop-project in the patch passed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
24s{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_121.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HADOOP-14040 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853105/HADOOP-14040-branch-2.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux 99c20f9d9889 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
[jira] [Commented] (HADOOP-14083) KMS should support old SSL clients
[
https://issues.apache.org/jira/browse/HADOOP-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870431#comment-15870431
]
Robert Kanter commented on HADOOP-14083:
In that case, we could have a branch-2 version of the patch which includes the
older ciphers for compatibility, and a trunk version of the patch that does not
for security. That said, we have broken compatibility in the past for security
fixes.
> KMS should support old SSL clients
> --
>
> Key: HADOOP-14083
> URL: https://issues.apache.org/jira/browse/HADOOP-14083
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.8.0, 2.7.4, 2.6.6
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-14083.branch-2.001.patch
>
>
> HADOOP-13812 upgraded Tomcat to 6.0.48 which filters weak ciphers. Old SSL
> clients such as curl stop working. The symptom is {{NSS error -12286}} when
> running {{curl -v}}.
> Instead of forcing the SSL clients to upgrade, we can configure Tomcat to
> explicitly allow enough weak ciphers so that old SSL clients can work.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14040) Use shaded aws-sdk uber-JAR 1.11.86
[
https://issues.apache.org/jira/browse/HADOOP-14040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870416#comment-15870416
]
Hadoop QA commented on HADOOP-14040:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
30s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
1s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
3s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 22s{color}
| {color:red} root generated 9 new + 699 unchanged - 1 fixed = 708 total (was
700) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
43s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14040 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853099/HADOOP-14040-HADOOP-13345.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux 4fa3f1bdbcb7 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / 94287ce |
| Default Java | 1.8.0_121 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11645/artifact/patchprocess/diff-compile-javac-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11645/testReport/ |
| modules | C: hadoop-project hadoop-tools/hadoop-aws U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11645/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Use shaded aws-sdk uber-JAR 1.11.86
> ---
>
> Key: HADOOP-14040
> URL: https://issues.apache.org/jira/browse/HADOOP-14040
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build,
[jira] [Commented] (HADOOP-14049) Honour AclBit flag associated to file/folder permission for Azure datalake account
[
https://issues.apache.org/jira/browse/HADOOP-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870388#comment-15870388
]
Hadoop QA commented on HADOOP-14049:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
27s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14049 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853107/HADOOP-14049.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5c97ea1ed43f 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5690b51 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11646/testReport/ |
| modules | C: hadoop-tools/hadoop-azure-datalake U:
hadoop-tools/hadoop-azure-datalake |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11646/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Honour AclBit flag associated to file/folder permission for Azure datalake
> account
> --
>
> Key: HADOOP-14049
> URL: https://issues.apache.org/jira/browse/HADOOP-14049
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: HADOOP-14049-01.patch, HADOOP-14049.02.patch
>
>
> ADLS persist AclBit information on a
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870362#comment-15870362 ] Steve Loughran commented on HADOOP-13075: - something that cant be fixed; no playing git games with the main branches. Just a note here in the JIRA for people to find if they look for it by way of the release notes, and if cherry picking > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Andrew Olson >Assignee: Steve Moist > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-13075-001.patch, HADOOP-13075-002.patch, > HADOOP-13075-003.patch, HADOOP-13075-branch2.002.patch > > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14049) Honour AclBit flag associated to file/folder permission for Azure datalake account
[
https://issues.apache.org/jira/browse/HADOOP-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishwajeet Dusane updated HADOOP-14049:
---
Status: Open (was: Patch Available)
> Honour AclBit flag associated to file/folder permission for Azure datalake
> account
> --
>
> Key: HADOOP-14049
> URL: https://issues.apache.org/jira/browse/HADOOP-14049
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: HADOOP-14049-01.patch, HADOOP-14049.02.patch
>
>
> ADLS persist AclBit information on a file/folder. Since Java SDK 2.1.4 -
> AclBit value can be retrieved using {{DirectoryEntry.aclBit}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14049) Honour AclBit flag associated to file/folder permission for Azure datalake account
[
https://issues.apache.org/jira/browse/HADOOP-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishwajeet Dusane updated HADOOP-14049:
---
Status: Patch Available (was: Open)
> Honour AclBit flag associated to file/folder permission for Azure datalake
> account
> --
>
> Key: HADOOP-14049
> URL: https://issues.apache.org/jira/browse/HADOOP-14049
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: HADOOP-14049-01.patch, HADOOP-14049.02.patch
>
>
> ADLS persist AclBit information on a file/folder. Since Java SDK 2.1.4 -
> AclBit value can be retrieved using {{DirectoryEntry.aclBit}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14049) Honour AclBit flag associated to file/folder permission for Azure datalake account
[
https://issues.apache.org/jira/browse/HADOOP-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishwajeet Dusane updated HADOOP-14049:
---
Attachment: HADOOP-14049.02.patch
Incorporated review comments from [~liuml07].
> Honour AclBit flag associated to file/folder permission for Azure datalake
> account
> --
>
> Key: HADOOP-14049
> URL: https://issues.apache.org/jira/browse/HADOOP-14049
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: HADOOP-14049-01.patch, HADOOP-14049.02.patch
>
>
> ADLS persist AclBit information on a file/folder. Since Java SDK 2.1.4 -
> AclBit value can be retrieved using {{DirectoryEntry.aclBit}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14040) Use shaded aws-sdk uber-JAR 1.11.86
[ https://issues.apache.org/jira/browse/HADOOP-14040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HADOOP-14040: --- Attachment: HADOOP-14040-branch-2-001.patch And this is my trunk patch applied to branch-2 with the conflicts resolved. All S3 tests pass on branch-2 also. > Use shaded aws-sdk uber-JAR 1.11.86 > --- > > Key: HADOOP-14040 > URL: https://issues.apache.org/jira/browse/HADOOP-14040 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14040.001.patch, HADOOP-14040-branch-2-001.patch, > HADOOP-14040-branch-2.002.patch, HADOOP-14040-HADOOP-13345.001.patch > > > AWS SDK now has a (v. large) uberjar shading all dependencies > This ensures that AWS dependency changes (e.g json) don't cause problems > downstream in things like HBase, so enabling backporting if desired. > This will let us addess the org.json don't be evil problem: this SDK version > doesn't have those files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14040) Use shaded aws-sdk uber-JAR 1.11.86
[ https://issues.apache.org/jira/browse/HADOOP-14040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HADOOP-14040: --- Attachment: (was: HADOOP-14040-branch-2-001.patch) > Use shaded aws-sdk uber-JAR 1.11.86 > --- > > Key: HADOOP-14040 > URL: https://issues.apache.org/jira/browse/HADOOP-14040 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14040.001.patch, HADOOP-14040-branch-2-001.patch, > HADOOP-14040-branch-2.002.patch, HADOOP-14040-HADOOP-13345.001.patch > > > AWS SDK now has a (v. large) uberjar shading all dependencies > This ensures that AWS dependency changes (e.g json) don't cause problems > downstream in things like HBase, so enabling backporting if desired. > This will let us addess the org.json don't be evil problem: this SDK version > doesn't have those files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14040) Use shaded aws-sdk uber-JAR 1.11.86
[ https://issues.apache.org/jira/browse/HADOOP-14040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HADOOP-14040: --- Attachment: HADOOP-14040-branch-2.002.patch > Use shaded aws-sdk uber-JAR 1.11.86 > --- > > Key: HADOOP-14040 > URL: https://issues.apache.org/jira/browse/HADOOP-14040 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14040.001.patch, HADOOP-14040-branch-2-001.patch, > HADOOP-14040-branch-2.002.patch, HADOOP-14040-HADOOP-13345.001.patch > > > AWS SDK now has a (v. large) uberjar shading all dependencies > This ensures that AWS dependency changes (e.g json) don't cause problems > downstream in things like HBase, so enabling backporting if desired. > This will let us addess the org.json don't be evil problem: this SDK version > doesn't have those files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14049) Honour AclBit flag associated to file/folder permission for Azure datalake account
[
https://issues.apache.org/jira/browse/HADOOP-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870315#comment-15870315
]
Vishwajeet Dusane commented on HADOOP-14049:
Thanks [~liuml07] for the review. I accept comment for 1. and 2., Will make the
change and upload 2nd iteration.
> Honour AclBit flag associated to file/folder permission for Azure datalake
> account
> --
>
> Key: HADOOP-14049
> URL: https://issues.apache.org/jira/browse/HADOOP-14049
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: HADOOP-14049-01.patch
>
>
> ADLS persist AclBit information on a file/folder. Since Java SDK 2.1.4 -
> AclBit value can be retrieved using {{DirectoryEntry.aclBit}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14040) Use shaded aws-sdk uber-JAR 1.11.86
[ https://issues.apache.org/jira/browse/HADOOP-14040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HADOOP-14040: --- Attachment: HADOOP-14040-HADOOP-13345.001.patch HADOOP-14040.001.patch I can at least attach a clean room implementation (at least as clean room as it can be having seen the patch a week ago - it's not like there's a ton of ways to do this that make sense) for trunk and the HADOOP-13345 branches. All tests passed against us-east-2. > Use shaded aws-sdk uber-JAR 1.11.86 > --- > > Key: HADOOP-14040 > URL: https://issues.apache.org/jira/browse/HADOOP-14040 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14040.001.patch, HADOOP-14040-branch-2-001.patch, > HADOOP-14040-HADOOP-13345.001.patch > > > AWS SDK now has a (v. large) uberjar shading all dependencies > This ensures that AWS dependency changes (e.g json) don't cause problems > downstream in things like HBase, so enabling backporting if desired. > This will let us addess the org.json don't be evil problem: this SDK version > doesn't have those files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14017) User friendly name for ADLS user and group
[
https://issues.apache.org/jira/browse/HADOOP-14017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870292#comment-15870292
]
Vishwajeet Dusane commented on HADOOP-14017:
{{chown}} and {{setfacl}} accepts user and group name as set in Azure AD tenant
when run against ADLS {{FileSystem}}. The change was required in ADLS store
back-end and is available in prod environment now.
> User friendly name for ADLS user and group
> --
>
> Key: HADOOP-14017
> URL: https://issues.apache.org/jira/browse/HADOOP-14017
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: John Zhuge
>Assignee: Vishwajeet Dusane
> Attachments: HADOOP-14017.01.patch
>
>
> ADLS displays GUID whenever user or group displayed, e.g., {{ls}},
> {{getfacl}}.
> ADLS requires GUID whenever user or group input is needed, e.g., {{setfacl}},
> {{chown}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870245#comment-15870245 ] Steve Moist commented on HADOOP-13075: -- Is that something that we need to fix? > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Andrew Olson >Assignee: Steve Moist > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-13075-001.patch, HADOOP-13075-002.patch, > HADOOP-13075-003.patch, HADOOP-13075-branch2.002.patch > > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13282) S3 blob etags to be made visible in status/getFileChecksum() calls
[
https://issues.apache.org/jira/browse/HADOOP-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13282:
Description:
If the etags of blobs were exported via {{getFileChecksum()}}, it'd be possible
to probe for a blob being in sync with a local file. Distcp could use this to
decide whether to skip a file or not.
Now, there's a problem there: distcp needs source and dest filesystems to
implement the same algorithm. It'd only work out the box if you were copying
between S3 instances. There are also quirks with encryption and multipart: [s3
docs|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html].
At the very least, it's something which could be used when indexing the FS, to
check for changes later.
was:
If the etags of blobs were exported via {{getFileChecksum() }}, it'd be
possible to probe for a blob being in sync with a local file. Distcp could use
this to decide whether to skip a file or not.
Now, there's a problem there: distcp needs source and dest filesystems to
implement the same algorithm. It'd only work out the box if you were copying
between S3 instances. There are also quirks with encryption and multipart: [s3
docs|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html].
At the very least, it's something which could be used when indexing the FS, to
check for changes later.
> S3 blob etags to be made visible in status/getFileChecksum() calls
> --
>
> Key: HADOOP-13282
> URL: https://issues.apache.org/jira/browse/HADOOP-13282
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
>
> If the etags of blobs were exported via {{getFileChecksum()}}, it'd be
> possible to probe for a blob being in sync with a local file. Distcp could
> use this to decide whether to skip a file or not.
> Now, there's a problem there: distcp needs source and dest filesystems to
> implement the same algorithm. It'd only work out the box if you were copying
> between S3 instances. There are also quirks with encryption and multipart:
> [s3
> docs|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html].
> At the very least, it's something which could be used when indexing the FS,
> to check for changes later.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13817) Add a finite shell command timeout to ShellBasedUnixGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870233#comment-15870233
]
Wei-Chiu Chuang commented on HADOOP-13817:
--
Hey [~qwertymaniac] this is a big patch, but sure I'll review it today.
> Add a finite shell command timeout to ShellBasedUnixGroupsMapping
> -
>
> Key: HADOOP-13817
> URL: https://issues.apache.org/jira/browse/HADOOP-13817
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.6.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Minor
>
> The ShellBasedUnixGroupsMapping run various {{id}} commands via the
> ShellCommandExecutor modules without a timeout set (its set to 0, which
> implies infinite).
> If this command hangs for a long time on the OS end due to an unresponsive
> groups backend or other reasons, it also blocks the handlers that use it on
> the NameNode (or other services that use this class). That inadvertently
> causes odd timeout troubles on the client end where its forced to retry (only
> to likely run into such hangs again with every attempt until at least one
> command returns).
> It would be helpful to have a finite command timeout after which we may give
> up on the command and return the result equivalent of no groups found.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14028) S3A block output streams don't delete temporary files in multipart uploads
[
https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870213#comment-15870213
]
Steve Loughran commented on HADOOP-14028:
-
Looking at the latest code (the stuff we get with HADOOP-14040). The check is
purely about mark supported and rollback
{code}
if (requestInputStream.markSupported()) {
try {
requestInputStream.reset();
} catch (IOException ex) {
throw new ResetException("Failed to reset the request
input stream", ex);
}
}
{code}
I will do a test to verify that all blocks are supporting reset of the full
datablock
> S3A block output streams don't delete temporary files in multipart uploads
> --
>
> Key: HADOOP-14028
> URL: https://issues.apache.org/jira/browse/HADOOP-14028
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: JDK 8 + ORC 1.3.0 + hadoop-aws 3.0.0-alpha2
>Reporter: Seth Fitzsimmons
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-14028-branch-2-001.patch,
> HADOOP-14028-branch-2.8-002.patch, HADOOP-14028-branch-2.8-003.patch,
> HADOOP-14028-branch-2.8-004.patch, HADOOP-14028-branch-2.8-005.patch
>
>
> I have `fs.s3a.fast.upload` enabled with 3.0.0-alpha2 (it's exactly what I
> was looking for after running into the same OOM problems) and don't see it
> cleaning up the disk-cached blocks.
> I'm generating a ~50GB file on an instance with ~6GB free when the process
> starts. My expectation is that local copies of the blocks would be deleted
> after those parts finish uploading, but I'm seeing more than 15 blocks in
> /tmp (and none of them have been deleted thus far).
> I see that DiskBlock deletes temporary files when closed, but is it closed
> after individual blocks have finished uploading or when the entire file has
> been fully written to the FS (full upload completed, including all parts)?
> As a temporary workaround to avoid running out of space, I'm listing files,
> sorting by atime, and deleting anything older than the first 20: `ls -ut |
> tail -n +21 | xargs rm`
> Steve Loughran says:
> > They should be deleted as soon as the upload completes; the close() call
> > that the AWS httpclient makes on the input stream triggers the deletion.
> > Though there aren't tests for it, as I recall.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14088) Java 8 javadoc errors when releasing
[ https://issues.apache.org/jira/browse/HADOOP-14088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870183#comment-15870183 ] Daniel Templeton commented on HADOOP-14088: --- If you can figure out a way to contribute the javadoc fixes, that would be awesome. Can you maybe create a diff patch from what you have and then just delete/revert the parts that aren't javadoc changes? > Java 8 javadoc errors when releasing > > > Key: HADOOP-14088 > URL: https://issues.apache.org/jira/browse/HADOOP-14088 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.5 > Environment: Java 8, CentOS 7 >Reporter: Serhiy Boychenko >Priority: Minor > Original Estimate: 8h > Remaining Estimate: 8h > > There are many javadoc errors popping up when releasing Hadoop. After doing > some modifications I have executed the following command to create a release: > mvn package -pdist -Psrc - Dtar -DskipTests > Many error regarding the usage of the self-closed tags (like ). Some > related to embedding into (which seems to be invalid). Many HTML > errors (unxesting tags, etc), problems related with generic representations > in javadoc which are being treated like tags, etc. Unfortunately I cannot > contribute my code since I have done some breaking changes, but I could > checkout again and try to fix errors (took me around 8 hours). There are also > loads of warnings, but at least those do not block release. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13321) Deprecate FileSystem APIs that promote inefficient call patterns.
[
https://issues.apache.org/jira/browse/HADOOP-13321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13321:
Status: Open (was: Patch Available)
patch doesn't apply any more im afraid, the javadoc improvements broke the
diff. If you can resync and resubmit it, I'll review it again
> Deprecate FileSystem APIs that promote inefficient call patterns.
> -
>
> Key: HADOOP-13321
> URL: https://issues.apache.org/jira/browse/HADOOP-13321
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
> Attachments: HADOOP-13321.000.patch, HADOOP-13321.001.patch
>
>
> {{FileSystem}} contains several methods that act as convenience wrappers over
> calling {{getFileStatus}} and retrieving a single property of the returned
> {{FileStatus}}. These methods have a habit of fostering inefficient call
> patterns in applications, resulting in multiple redundant {{getFileStatus}}
> calls. For HDFS, this translates into wasteful NameNode RPC traffic. For
> file systems backed by cloud object stores, this translates into wasteful
> HTTP traffic. This issue proposes to deprecate these methods and instead
> encourage applications to call {{getFileStatus}} and then reuse the same
> {{FileStatus}} instance as needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13694) Data transfer encryption with AES 192: Invalid key length.
[ https://issues.apache.org/jira/browse/HADOOP-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870061#comment-15870061 ] Harsh J commented on HADOOP-13694: -- [~jojochuang] - Could you help in reviewing this? > Data transfer encryption with AES 192: Invalid key length. > -- > > Key: HADOOP-13694 > URL: https://issues.apache.org/jira/browse/HADOOP-13694 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.7.2 > Environment: OS: Ubuntu 14.04 > /hadoop-2.7.2/bin$ uname -a > Linux wkstn-kpalaniappan 3.13.0-79-generic #123-Ubuntu SMP Fri Feb 19 > 14:27:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux > /hadoop-2.7.2/bin$ java -version > java version "1.7.0_95" > OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-0ubuntu0.14.04.1) > OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode) > Hadoop version: 2.7.2 >Reporter: Karthik Palaniappan >Assignee: Harsh J > > Configuring aes 128 or aes 256 encryption > (dfs.encrypt.data.transfer.cipher.key.bitlength = [128, 256]) works perfectly > fine. Trying to use AES 192 generates this exception on the datanode: > 16/02/29 17:34:10 ERROR datanode.DataNode: > wkstn-kpalaniappan:50010:DataXceiver error processing unknown operation src: > /127.0.0.1:57237 dst: /127.0.0.1:50010 > java.lang.IllegalArgumentException: Invalid key length. > at org.apache.hadoop.crypto.OpensslCipher.init(Native Method) > at org.apache.hadoop.crypto.OpensslCipher.init(OpensslCipher.java:176) > at > org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec$OpensslAesCtrCipher.init(OpensslAesCtrCryptoCodec.java:116) > at > org.apache.hadoop.crypto.CryptoInputStream.updateDecryptor(CryptoInputStream.java:290) > at > org.apache.hadoop.crypto.CryptoInputStream.resetStreamOffset(CryptoInputStream.java:303) > at > org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:128) > at > org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:109) > at > org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:133) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.createStreamPair(DataTransferSaslUtil.java:345) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:396) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:178) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:110) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:193) > at java.lang.Thread.run(Thread.java:745) > And this exception on the client: > /hadoop-2.7.2/bin$ ./hdfs dfs -copyFromLocal ~/.vimrc /vimrc > 16/02/29 17:34:10 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Invalid key length. > at org.apache.hadoop.crypto.OpensslCipher.init(Native Method) > at org.apache.hadoop.crypto.OpensslCipher.init(OpensslCipher.java:176) > at > org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec$OpensslAesCtrCipher.init(OpensslAesCtrCryptoCodec.java:116) > at > org.apache.hadoop.crypto.CryptoInputStream.updateDecryptor(CryptoInputStream.java:290) > at > org.apache.hadoop.crypto.CryptoInputStream.resetStreamOffset(CryptoInputStream.java:303) > at > org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:128) > at > org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:109) > at > org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:133) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.createStreamPair(DataTransferSaslUtil.java:345) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:490) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:299) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:242) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211) > at > org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1266) > at >
[jira] [Commented] (HADOOP-13817) Add a finite shell command timeout to ShellBasedUnixGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870058#comment-15870058
]
Harsh J commented on HADOOP-13817:
--
[~jojochuang] - Could you help in reviewing this? Current diff is at
https://github.com/apache/hadoop/pull/161
> Add a finite shell command timeout to ShellBasedUnixGroupsMapping
> -
>
> Key: HADOOP-13817
> URL: https://issues.apache.org/jira/browse/HADOOP-13817
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.6.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Minor
>
> The ShellBasedUnixGroupsMapping run various {{id}} commands via the
> ShellCommandExecutor modules without a timeout set (its set to 0, which
> implies infinite).
> If this command hangs for a long time on the OS end due to an unresponsive
> groups backend or other reasons, it also blocks the handlers that use it on
> the NameNode (or other services that use this class). That inadvertently
> causes odd timeout troubles on the client end where its forced to retry (only
> to likely run into such hangs again with every attempt until at least one
> command returns).
> It would be helpful to have a finite command timeout after which we may give
> up on the command and return the result equivalent of no groups found.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13321) Deprecate FileSystem APIs that promote inefficient call patterns.
[
https://issues.apache.org/jira/browse/HADOOP-13321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870047#comment-15870047
]
Hadoop QA commented on HADOOP-13321:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HADOOP-13321 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-13321 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12825714/HADOOP-13321.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11644/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Deprecate FileSystem APIs that promote inefficient call patterns.
> -
>
> Key: HADOOP-13321
> URL: https://issues.apache.org/jira/browse/HADOOP-13321
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
> Attachments: HADOOP-13321.000.patch, HADOOP-13321.001.patch
>
>
> {{FileSystem}} contains several methods that act as convenience wrappers over
> calling {{getFileStatus}} and retrieving a single property of the returned
> {{FileStatus}}. These methods have a habit of fostering inefficient call
> patterns in applications, resulting in multiple redundant {{getFileStatus}}
> calls. For HDFS, this translates into wasteful NameNode RPC traffic. For
> file systems backed by cloud object stores, this translates into wasteful
> HTTP traffic. This issue proposes to deprecate these methods and instead
> encourage applications to call {{getFileStatus}} and then reuse the same
> {{FileStatus}} instance as needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13321) Deprecate FileSystem APIs that promote inefficient call patterns.
[
https://issues.apache.org/jira/browse/HADOOP-13321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870025#comment-15870025
]
Steve Loughran commented on HADOOP-13321:
-
+1 for this on trunk
> Deprecate FileSystem APIs that promote inefficient call patterns.
> -
>
> Key: HADOOP-13321
> URL: https://issues.apache.org/jira/browse/HADOOP-13321
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
> Attachments: HADOOP-13321.000.patch, HADOOP-13321.001.patch
>
>
> {{FileSystem}} contains several methods that act as convenience wrappers over
> calling {{getFileStatus}} and retrieving a single property of the returned
> {{FileStatus}}. These methods have a habit of fostering inefficient call
> patterns in applications, resulting in multiple redundant {{getFileStatus}}
> calls. For HDFS, this translates into wasteful NameNode RPC traffic. For
> file systems backed by cloud object stores, this translates into wasteful
> HTTP traffic. This issue proposes to deprecate these methods and instead
> encourage applications to call {{getFileStatus}} and then reuse the same
> {{FileStatus}} instance as needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14040) Use shaded aws-sdk uber-JAR 1.11.86
[ https://issues.apache.org/jira/browse/HADOOP-14040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870017#comment-15870017 ] Steve Loughran commented on HADOOP-14040: - thanks, afraid I still need someone with branch-2 commit rights to get it in. Anyone else willing to vote on it? Or I can cancel my patch, sean can do a clean room reimplementation of it and then I get to vote on that myself? > Use shaded aws-sdk uber-JAR 1.11.86 > --- > > Key: HADOOP-14040 > URL: https://issues.apache.org/jira/browse/HADOOP-14040 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14040-branch-2-001.patch > > > AWS SDK now has a (v. large) uberjar shading all dependencies > This ensures that AWS dependency changes (e.g json) don't cause problems > downstream in things like HBase, so enabling backporting if desired. > This will let us addess the org.json don't be evil problem: this SDK version > doesn't have those files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14040) Use shaded aws-sdk uber-JAR 1.11.86
[ https://issues.apache.org/jira/browse/HADOOP-14040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14040: Summary: Use shaded aws-sdk uber-JAR 1.11.86 (was: Use shaded aws-sdk uber-JAR) > Use shaded aws-sdk uber-JAR 1.11.86 > --- > > Key: HADOOP-14040 > URL: https://issues.apache.org/jira/browse/HADOOP-14040 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14040-branch-2-001.patch > > > AWS SDK now has a (v. large) uberjar shading all dependencies > This ensures that AWS dependency changes (e.g json) don't cause problems > downstream in things like HBase, so enabling backporting if desired. > This will let us addess the org.json don't be evil problem: this SDK version > doesn't have those files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14088) Java 8 javadoc errors when releasing
[ https://issues.apache.org/jira/browse/HADOOP-14088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Serhiy Boychenko updated HADOOP-14088: -- Summary: Java 8 javadoc errors when releasing (was: Java 8 javadoc error when releasing) > Java 8 javadoc errors when releasing > > > Key: HADOOP-14088 > URL: https://issues.apache.org/jira/browse/HADOOP-14088 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.5 > Environment: Java 8, CentOS 7 >Reporter: Serhiy Boychenko >Priority: Minor > Original Estimate: 8h > Remaining Estimate: 8h > > There are many javadoc errors popping up when releasing Hadoop. After doing > some modifications I have executed the following command to create a release: > mvn package -pdist -Psrc - Dtar -DskipTests > Many error regarding the usage of the self-closed tags (like ). Some > related to embedding into (which seems to be invalid). Many HTML > errors (unxesting tags, etc), problems related with generic representations > in javadoc which are being treated like tags, etc. Unfortunately I cannot > contribute my code since I have done some breaking changes, but I could > checkout again and try to fix errors (took me around 8 hours). There are also > loads of warnings, but at least those do not block release. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14088) Java 8 javadoc error when releasing
[ https://issues.apache.org/jira/browse/HADOOP-14088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Serhiy Boychenko updated HADOOP-14088: -- Description: There are many javadoc errors popping up when releasing Hadoop. After doing some modifications I have executed the following command to create a release: mvn package -pdist -Psrc - Dtar -DskipTests Many error regarding the usage of the self-closed tags (like ). Some related to embedding into . Many HTML errors (unxesting tags, etc), problems related with generic representations in javadoc which are being treated like tags, etc. Unfortunately I cannot contribute my code since I have done some breaking changes, but I could checkout again and try to fix errors (took me around 8 hours). There are also loads of warnings, but at least those do not block release. was: There are many javadoc errors popping up when releasing Hadoop. After doing some modifications I have executed the following command to create a release: mvn package -pdist -Psrc - Dtar -DskipTests Many error regarding the usage of the self-closed tags (like ). Some related to embedding into . Many HTML errors (unxesting tags, etc), problems related with generic representations in javadoc which are being treated like tags, etc. Unfortunately I cannot contribute my build since I have done some breaking changes, but I could checkout again and try to fix errors (took me around 8 hours). There are also loads of warnings, but at least those do not block release. > Java 8 javadoc error when releasing > --- > > Key: HADOOP-14088 > URL: https://issues.apache.org/jira/browse/HADOOP-14088 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.5 > Environment: Java 8, CentOS 7 >Reporter: Serhiy Boychenko >Priority: Minor > Original Estimate: 8h > Remaining Estimate: 8h > > There are many javadoc errors popping up when releasing Hadoop. After doing > some modifications I have executed the following command to create a release: > mvn package -pdist -Psrc - Dtar -DskipTests > Many error regarding the usage of the self-closed tags (like ). Some > related to embedding into . Many HTML errors (unxesting tags, etc), > problems related with generic representations in javadoc which are being > treated like tags, etc. Unfortunately I cannot contribute my code since I > have done some breaking changes, but I could checkout again and try to fix > errors (took me around 8 hours). There are also loads of warnings, but at > least those do not block release. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14088) Java 8 javadoc error when releasing
[ https://issues.apache.org/jira/browse/HADOOP-14088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Serhiy Boychenko updated HADOOP-14088: -- Description: There are many javadoc errors popping up when releasing Hadoop. After doing some modifications I have executed the following command to create a release: mvn package -pdist -Psrc - Dtar -DskipTests Many error regarding the usage of the self-closed tags (like ). Some related to embedding into (which seems to be invalid). Many HTML errors (unxesting tags, etc), problems related with generic representations in javadoc which are being treated like tags, etc. Unfortunately I cannot contribute my code since I have done some breaking changes, but I could checkout again and try to fix errors (took me around 8 hours). There are also loads of warnings, but at least those do not block release. was: There are many javadoc errors popping up when releasing Hadoop. After doing some modifications I have executed the following command to create a release: mvn package -pdist -Psrc - Dtar -DskipTests Many error regarding the usage of the self-closed tags (like ). Some related to embedding into . Many HTML errors (unxesting tags, etc), problems related with generic representations in javadoc which are being treated like tags, etc. Unfortunately I cannot contribute my code since I have done some breaking changes, but I could checkout again and try to fix errors (took me around 8 hours). There are also loads of warnings, but at least those do not block release. > Java 8 javadoc error when releasing > --- > > Key: HADOOP-14088 > URL: https://issues.apache.org/jira/browse/HADOOP-14088 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.5 > Environment: Java 8, CentOS 7 >Reporter: Serhiy Boychenko >Priority: Minor > Original Estimate: 8h > Remaining Estimate: 8h > > There are many javadoc errors popping up when releasing Hadoop. After doing > some modifications I have executed the following command to create a release: > mvn package -pdist -Psrc - Dtar -DskipTests > Many error regarding the usage of the self-closed tags (like ). Some > related to embedding into (which seems to be invalid). Many HTML > errors (unxesting tags, etc), problems related with generic representations > in javadoc which are being treated like tags, etc. Unfortunately I cannot > contribute my code since I have done some breaking changes, but I could > checkout again and try to fix errors (took me around 8 hours). There are also > loads of warnings, but at least those do not block release. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14088) Java 8 javadoc error when releasing
Serhiy Boychenko created HADOOP-14088: - Summary: Java 8 javadoc error when releasing Key: HADOOP-14088 URL: https://issues.apache.org/jira/browse/HADOOP-14088 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.6.5 Environment: Java 8, CentOS 7 Reporter: Serhiy Boychenko Priority: Minor There are many javadoc errors popping up when releasing Hadoop. After doing some modifications I have executed the following command to create a release: mvn package -pdist -Psrc - Dtar -DskipTests Many error regarding the usage of the self-closed tags (like ). Some related to embedding into . Many HTML errors (unxesting tags, etc), problems related with generic representations in javadoc which are being treated like tags, etc. Unfortunately I cannot contribute my build since I have done some breaking changes, but I could checkout again and try to fix errors (took me around 8 hours). There are also loads of warnings, but at least those do not block release. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13973) S3A get requests failing: java.lang.IllegalStateException: Connection is not open
[ https://issues.apache.org/jira/browse/HADOOP-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869924#comment-15869924 ] Steve Loughran commented on HADOOP-13973: - Later httpclients move to ConnectionClosedException. A cause is possibly/probably us calling abort() on a connection which is still in the pool, and so being picked up again > S3A get requests failing: java.lang.IllegalStateException: Connection is not > open > - > > Key: HADOOP-13973 > URL: https://issues.apache.org/jira/browse/HADOOP-13973 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: EC2 cluster >Reporter: Rajesh Balamohan >Assignee: Steve Loughran > Fix For: 2.8.0 > > > S3 requests failing with an error coming from Http client, > "java.lang.IllegalStateException: Connection is not open" > Some online discussion implies that this is related to shared connection pool > shutdown & fixed in http client 4.4+. Hadoop & AWS SDK use v 4.5.2 so the fix > is in, we just need to make sure the pool is being set up right. > There's a problem here of course: it may require moving to a later version of > the AWS SDK, with the consequences on jackson , as seen in HADOOP-13050. > And that's if there is a patched version out there -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13973) S3A get requests failing: java.lang.IllegalStateException: Connection is not open
[ https://issues.apache.org/jira/browse/HADOOP-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13973: Summary: S3A get requests failing: java.lang.IllegalStateException: Connection is not open (was: S3 requests failing: java.lang.IllegalStateException: Connection is not open) > S3A get requests failing: java.lang.IllegalStateException: Connection is not > open > - > > Key: HADOOP-13973 > URL: https://issues.apache.org/jira/browse/HADOOP-13973 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: EC2 cluster >Reporter: Rajesh Balamohan >Assignee: Steve Loughran > Fix For: 2.8.0 > > > S3 requests failing with an error coming from Http client, > "java.lang.IllegalStateException: Connection is not open" > Some online discussion implies that this is related to shared connection pool > shutdown & fixed in http client 4.4+. Hadoop & AWS SDK use v 4.5.2 so the fix > is in, we just need to make sure the pool is being set up right. > There's a problem here of course: it may require moving to a later version of > the AWS SDK, with the consequences on jackson , as seen in HADOOP-13050. > And that's if there is a patched version out there -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869816#comment-15869816 ] Steve Loughran commented on HADOOP-13075: - just changed the title to put the word "encryption" in. This patch was committed with the wrong JIRA number BTW: HADOOP-13204 > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Andrew Olson >Assignee: Steve Moist > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-13075-001.patch, HADOOP-13075-002.patch, > HADOOP-13075-003.patch, HADOOP-13075-branch2.002.patch > > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-11694) Über-jira: S3a phase II: robustness, scale and performance
[ https://issues.apache.org/jira/browse/HADOOP-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reopened HADOOP-11694: - assuming this covers all for 2.8, there are some late-breaking bugs coming up..reopening > Über-jira: S3a phase II: robustness, scale and performance > -- > > Key: HADOOP-11694 > URL: https://issues.apache.org/jira/browse/HADOOP-11694 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.7.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 2.8.0 > > > HADOOP-11571 covered the core s3a bugs surfacing in Hadoop-2.6 & other > enhancements to improve S3 (performance, proxy, custom endpoints) > This JIRA covers post-2.7 issues and enhancements. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14081) S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock)
[
https://issues.apache.org/jira/browse/HADOOP-14081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869810#comment-15869810
]
Steve Loughran commented on HADOOP-14081:
-
# as usual, please state which s3 region you tested with. As this plays with
blocks, please use -Dscale for the scale tests, ideally choosing a larger file
size for more rigorousnes
# This patch will break against HADOOP-14208, whichever gets in first will
force the other to catch up. That's a bug so has priority for me, but this is a
lower cost patch which I can vote in. So: once you confirm the testing, I'll
take a look at this
> S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock)
> --
>
> Key: HADOOP-14081
> URL: https://issues.apache.org/jira/browse/HADOOP-14081
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HADOOP-14081.001.patch
>
>
> In {{S3ADataBlocks::ByteArrayBlock}}, data is copied whenever {{startUpload}}
> is called. It might be possible to directly access the byte[] array from
> ByteArrayOutputStream.
> Might have to extend ByteArrayOutputStream and create a method like
> getInputStream() which can return ByteArrayInputStream. This would avoid
> expensive array copy during large upload.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14076) Allow Configuration to be persisted given path to file
[
https://issues.apache.org/jira/browse/HADOOP-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869738#comment-15869738
]
Steve Loughran commented on HADOOP-14076:
-
OR we have some utils.ConfIO class with methods like {{saveToFile(File,
Configuration) and saveToLocalPath(string, Configuration) as well as the same
for Filesystem URIs. It's just that Configuration is big enough as it is, and
these things don't have to be added to that class as it is. But yes, chris has
a point too: do it outside of the core and your app gets to use it immediately,
no need to backport
> Allow Configuration to be persisted given path to file
> --
>
> Key: HADOOP-14076
> URL: https://issues.apache.org/jira/browse/HADOOP-14076
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ted Yu
>
> Currently Configuration has the following methods for persistence:
> {code}
> public void writeXml(OutputStream out) throws IOException {
> public void writeXml(Writer out) throws IOException {
> {code}
> Adding API for persisting to file given path would be useful:
> {code}
> public void writeXml(String path) throws IOException {
> {code}
> Background: I recently worked on exporting Configuration to a file using JNI.
> Without the proposed API, I resorted to some trick such as the following:
> http://www.kfu.com/~nsayer/Java/jni-filedesc.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13826) S3A Deadlock in multipart copy due to thread pool limits.
[
https://issues.apache.org/jira/browse/HADOOP-13826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869725#comment-15869725
]
Steve Loughran commented on HADOOP-13826:
-
thx for the reminder; forgotten that it was ready for review. Been laptopless
for a week, so still recovering. I'll try for the end of the week
> S3A Deadlock in multipart copy due to thread pool limits.
> -
>
> Key: HADOOP-13826
> URL: https://issues.apache.org/jira/browse/HADOOP-13826
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
>Priority: Critical
> Attachments: HADOOP-13826.001.patch, HADOOP-13826.002.patch,
> HADOOP-13826.003.patch, HADOOP-13826.004.patch
>
>
> In testing HIVE-15093 we have encountered deadlocks in the s3a connector. The
> TransferManager javadocs
> (http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html)
> explain how this is possible:
> {quote}It is not recommended to use a single threaded executor or a thread
> pool with a bounded work queue as control tasks may submit subtasks that
> can't complete until all sub tasks complete. Using an incorrectly configured
> thread pool may cause a deadlock (I.E. the work queue is filled with control
> tasks that can't finish until subtasks complete but subtasks can't execute
> because the queue is filled).{quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14086) Improve DistCp Speed for small files
[ https://issues.apache.org/jira/browse/HADOOP-14086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869708#comment-15869708 ] Steve Loughran commented on HADOOP-14086: - # target version will have to be branch-2+, with backports as people feel appropriate # please don't make things worse for object stores. One thing we've started doing there is massively boost the performance of listFiles(path, recursive=true), which we can take from being a slow emulation of a recursive treewalk to an O(1+ files/5000) call. If you could use that to iterate over the LocatedFileStatus entries, then hand off that status data direct to the workers, then it'd be great for object stores, while still delivering good NN perf > Improve DistCp Speed for small files > > > Key: HADOOP-14086 > URL: https://issues.apache.org/jira/browse/HADOOP-14086 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/distcp >Affects Versions: 2.6.5 >Reporter: Zheng Shao >Assignee: Zheng Shao >Priority: Minor > > When using distcp to copy lots of small files, NameNode naturally becomes a > bottleneck. > The current distcp code did *not* optimize to reduce the NameNode calls. We > should restructure the code to reduce the number of NameNode calls as much as > possible to speed up the copy of small files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14071) S3a: Failed to reset the request input stream
[
https://issues.apache.org/jira/browse/HADOOP-14071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869635#comment-15869635
]
Seth Fitzsimmons commented on HADOOP-14071:
---
For sure.
> S3a: Failed to reset the request input stream
> -
>
> Key: HADOOP-14071
> URL: https://issues.apache.org/jira/browse/HADOOP-14071
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.0.0-alpha2
>Reporter: Seth Fitzsimmons
>
> When using the patch from HADOOP-14028, I fairly consistently get {{Failed to
> reset the request input stream}} exceptions. They're more likely to occur the
> larger the file that's being written (70GB in the extreme case, but it needs
> to be one file).
> {code}
> 2017-02-10 04:21:43 WARN S3ABlockOutputStream:692 - Transfer failure of block
> FileBlock{index=416,
> destFile=/tmp/hadoop-root/s3a/s3ablock-0416-4228067786955989475.tmp,
> state=Upload, dataSize=11591473, limit=104857600}
> 2017-02-10 04:21:43 WARN S3AInstrumentation:777 - Closing output stream
> statistics while data is still marked as pending upload in
> OutputStreamStatistics{blocksSubmitted=416, blocksInQueue=0, blocksActive=0,
> blockUploadsCompleted=416, blockUploadsFailed=3,
> bytesPendingUpload=209747761, bytesUploaded=43317747712, blocksAllocated=416,
> blocksReleased=416, blocksActivelyAllocated=0,
> exceptionsInMultipartFinalize=0, transferDuration=1389936 ms,
> queueDuration=519 ms, averageQueueTime=1 ms, totalUploadDuration=1390455 ms,
> effectiveBandwidth=3.1153649497466657E7 bytes/s}
> at org.apache.hadoop.fs.s3a.S3AUtils.extractException(S3AUtils.java:200)
> at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:128)
> Exception in thread "main" org.apache.hadoop.fs.s3a.AWSClientIOException:
> Multi-part upload with id
> 'Xx.ezqT5hWrY1W92GrcodCip88i8rkJiOcom2nuUAqHtb6aQX__26FYh5uYWKlRNX5vY5ktdmQWlOovsbR8CLmxUVmwFkISXxDRHeor8iH9nPhI3OkNbWJJBLrvB3xLUuLX0zvGZWo7bUrAKB6IGxA--'
> to 2017/planet-170206.orc on 2017/planet-170206.orc:
> com.amazonaws.ResetException: Failed to reset the request input stream; If
> the request involves an input stream, the maximum stream buffer size can be
> configured via request.getRequestClientOptions().setReadLimit(int): Failed to
> reset the request input stream; If the request involves an input stream, the
> maximum stream buffer size can be configured via
> request.getRequestClientOptions().setReadLimit(int)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.waitForAllPartUploads(S3ABlockOutputStream.java:539)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.access$100(S3ABlockOutputStream.java:456)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:351)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at org.apache.orc.impl.PhysicalFsWriter.close(PhysicalFsWriter.java:221)
> at org.apache.orc.impl.WriterImpl.close(WriterImpl.java:2827)
> at net.mojodna.osm2orc.standalone.OsmPbf2Orc.convert(OsmPbf2Orc.java:296)
> at net.mojodna.osm2orc.Osm2Orc.main(Osm2Orc.java:47)
> Caused by: com.amazonaws.ResetException: Failed to reset the request input
> stream; If the request involves an input stream, the maximum stream buffer
> size can be configured via request.getRequestClientOptions().setReadLimit(int)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.resetRequestInputStream(AmazonHttpClient.java:1221)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1042)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
> at
> org.apache.hadoop.fs.s3a.SemaphoredDelegatingExecutor$CallableWithPermitRelease.call(SemaphoredDelegatingExecutor.java:222)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4041)
> at
> com.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3041)
> at
> com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3026)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.uploadPart(S3AFileSystem.java:1114)
> at
>
[jira] [Commented] (HADOOP-14083) KMS should support old SSL clients
[
https://issues.apache.org/jira/browse/HADOOP-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869544#comment-15869544
]
Xiao Chen commented on HADOOP-14083:
Thanks John for filing a jira and providing a patch, and Allen for discussion.
I agree with Allen that best practice is default to strong, and allow people to
configure.
But from this
[comment|https://issues.apache.org/jira/browse/HADOOP-13812?focusedCommentId=15695443=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15695443]
of HADOOP-13812, clients could break outright after upgrading. HADOOP-13812 is
marked incompatible, but in x.y.z branches to include tomcat security fixes.
So choosing between the two frown-upon's, IMO we should trade off for
compatibility here, and release doc it so security-concerned users are aware.
> KMS should support old SSL clients
> --
>
> Key: HADOOP-14083
> URL: https://issues.apache.org/jira/browse/HADOOP-14083
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.8.0, 2.7.4, 2.6.6
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-14083.branch-2.001.patch
>
>
> HADOOP-13812 upgraded Tomcat to 6.0.48 which filters weak ciphers. Old SSL
> clients such as curl stop working. The symptom is {{NSS error -12286}} when
> running {{curl -v}}.
> Instead of forcing the SSL clients to upgrade, we can configure Tomcat to
> explicitly allow enough weak ciphers so that old SSL clients can work.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14037) client.handleSaslConnectionFailure needlessly wraps IOEs
[
https://issues.apache.org/jira/browse/HADOOP-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869484#comment-15869484
]
Hadoop QA commented on HADOOP-14037:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 18s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestKDiag |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14037 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12852995/HADOOP-14037.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux de062be69962 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6c25dbc |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11643/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11643/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11643/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> client.handleSaslConnectionFailure needlessly wraps IOEs
>
>
> Key: HADOOP-14037
> URL: https://issues.apache.org/jira/browse/HADOOP-14037
> Project: Hadoop Common
>
