[jira] [Commented] (HADOOP-15426) Make S3guard client resilient to DDB throttle events and network failures
[
https://issues.apache.org/jira/browse/HADOOP-15426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598236#comment-16598236
]
genericqa commented on HADOOP-15426:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 56s{color} | {color:orange} root: The patch generated 15 new + 32 unchanged
- 2 fixed = 47 total (was 34) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 23 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 56s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
6s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
34s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15426 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12937784/HADOOP-15426-008.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 1b840870a225 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Updated] (HADOOP-15663) ABFS: Simplify configuration
[ https://issues.apache.org/jira/browse/HADOOP-15663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt updated HADOOP-15663: -- Resolution: Fixed Status: Resolved (was: Patch Available) commit d58ea7c96abe1dbba2f89019676a2dd869e6b550 Author: Thomas Marquardt Date: Fri Aug 31 03:24:42 2018 + HADOOP-15663. ABFS: Simplify configuration. Contributed by Da Zhou. > ABFS: Simplify configuration > > > Key: HADOOP-15663 > URL: https://issues.apache.org/jira/browse/HADOOP-15663 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-15663-HADOOP-15407-001.patch, > HADOOP-15663-HADOOP-15407-002.patch, HADOOP-15663-HADOOP-15407-003.patch, > HADOOP-15663-HADOOP-15407-004.patch, HADOOP-15663-HADOOP-15407-005.patch > > > Configuration for WASB and ABFS is too complex. The current approach is to > use four files for test configuration. > Both WASB and ABFS have basic test configuration which is committed to the > repo (azure-test.xml and azure-bfs-test.xml). Currently these contain the > fs.AbstractFileSystem.[scheme].impl configuration, but otherwise are empty > except for an include reference to a file containing the endpoint > credentials. > Both WASB and ABFS have endpoint credential configuration files > (azure-auth-keys.xml and azure-bfs-auth-keys.xml). These have been added to > .gitignore to prevent them from accidentally being submitted in a patch, > which would leak the developers storage account credentials. These files > contain account names, storage account keys, and service endpoints. > There is some overlap of the configuration for WASB and ABFS, where they use > the same property name but use different values. > 1) Let's reduce the number of test configuration files to one, if possible. > 2) Let's simplify the account name, key, and endpoint configuration for WASB > and ABFS if possible, but still support the legacy way of doing it, which is > very error prone. > 3) Let's improve error handling, so that typos or misconfiguration are not so > difficult to troubleshoot. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15663) ABFS: Simplify configuration
[
https://issues.apache.org/jira/browse/HADOOP-15663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598179#comment-16598179
]
Thomas Marquardt commented on HADOOP-15663:
---
+1, thanks for addressing the feedback. All tests pass against my US storage
account:
*mvn -T 1C -Dparallel-tests -Dscale -DtestsThreadCount=8 clean verify*
*Tests run: 265, Failures: 0, Errors: 0, Skipped: 11*
*Tests run: 1, Failures: 0, Errors: 0, Skipped: 0*
*Tests run: 863, Failures: 0, Errors: 0, Skipped: 262*
*Tests run: 186, Failures: 0, Errors: 0, Skipped: 10*
I made edits to testing_azure.md to clarify the configuration for running the
tests.
There is a section named *"Testing the Azure ABFS Client"* near the end of the
document,
and readers can copy paste the contents to the file
*src/test/resources/azure-auth-keys.xml*
to get started. The necessary contents are also below:
{noformat}
http://www.w3.org/2001/XInclude;>
fs.azure.abfs.account.name
{ACCOUNT_NAME}.dfs.core.windows.net
fs.azure.account.key.{ACCOUNT_NAME}.dfs.core.windows.net
{ACCOUNT_ACCESS_KEY}
fs.azure.wasb.account.name
{ACCOUNT_NAME}.blob.core.windows.net
fs.azure.account.key.{ACCOUNT_NAME}.blob.core.windows.net
{ACCOUNT_ACCESS_KEY}
fs.contract.test.fs.abfs
abfs://{CONTAINER_NAME}@{ACCOUNT_NAME}.dfs.core.windows.net
A file system URI to be used by the contract
tests.
fs.contract.test.fs.wasb
wasb://{CONTAINER_NAME}@{ACCOUNT_NAME}.blob.core.windows.net
A file system URI to be used by the contract
tests.
{noformat}
> ABFS: Simplify configuration
>
>
> Key: HADOOP-15663
> URL: https://issues.apache.org/jira/browse/HADOOP-15663
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15663-HADOOP-15407-001.patch,
> HADOOP-15663-HADOOP-15407-002.patch, HADOOP-15663-HADOOP-15407-003.patch,
> HADOOP-15663-HADOOP-15407-004.patch, HADOOP-15663-HADOOP-15407-005.patch
>
>
> Configuration for WASB and ABFS is too complex. The current approach is to
> use four files for test configuration.
> Both WASB and ABFS have basic test configuration which is committed to the
> repo (azure-test.xml and azure-bfs-test.xml). Currently these contain the
> fs.AbstractFileSystem.[scheme].impl configuration, but otherwise are empty
> except for an include reference to a file containing the endpoint
> credentials.
> Both WASB and ABFS have endpoint credential configuration files
> (azure-auth-keys.xml and azure-bfs-auth-keys.xml). These have been added to
> .gitignore to prevent them from accidentally being submitted in a patch,
> which would leak the developers storage account credentials. These files
> contain account names, storage account keys, and service endpoints.
> There is some overlap of the configuration for WASB and ABFS, where they use
> the same property name but use different values.
> 1) Let's reduce the number of test configuration files to one, if possible.
> 2) Let's simplify the account name, key, and endpoint configuration for WASB
> and ABFS if possible, but still support the legacy way of doing it, which is
> very error prone.
> 3) Let's improve error handling, so that typos or misconfiguration are not so
> difficult to troubleshoot.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15107) Stabilize/tune S3A committers; review correctness & docs
[ https://issues.apache.org/jira/browse/HADOOP-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598148#comment-16598148 ] Hudson commented on HADOOP-15107: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14855 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14855/]) HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs. (stevel: rev 5a0babf76550f63dad4c17173c4da2bf335c6532) * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Invoker.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/S3ACommitterFactory.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/staging/integration/ITestStagingCommitProtocol.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PathOutputCommitter.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/PartitionedStagingCommitter.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/magic/ITestMagicCommitProtocol.java * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/DirectoryStagingCommitter.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/magic/ITMagicCommitMRJob.java * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/staging/integration/ITStagingCommitMRJobBadDest.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/AbstractCommitITest.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/Paths.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/AbstractITCommitMRJob.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committer_architecture.md * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/StagingCommitter.java * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestS3ACommitterFactory.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/AbstractITCommitProtocol.java > Stabilize/tune S3A committers; review correctness & docs > > > Key: HADOOP-15107 > URL: https://issues.apache.org/jira/browse/HADOOP-15107 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Fix For: 3.1.2 > > Attachments: HADOOP-15107-001.patch, HADOOP-15107-002.patch, > HADOOP-15107-003.patch, HADOOP-15107-004.patch > > > I'm writing about the paper on the committers, one which, being a proper > paper, requires me to show the committers work. > # define the requirements of a "Correct" committed job (this applies to the > FileOutputCommitter too) > # show that the Staging committer meets these requirements (most of this is > implicit in that it uses the V1 FileOutputCommitter to marshall .pendingset > lists from committed tasks to the final destination, where they are read and > committed. > # Show the magic committer also works. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15680) ITestNativeAzureFileSystemConcurrencyLive times out
[ https://issues.apache.org/jira/browse/HADOOP-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598147#comment-16598147 ] Hudson commented on HADOOP-15680: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14855 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14855/]) HADOOP-15680. ITestNativeAzureFileSystemConcurrencyLive times out. (stevel: rev e8d138ca7c1b695688515d816ac693437c87df62) * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/ITestNativeAzureFileSystemConcurrencyLive.java > ITestNativeAzureFileSystemConcurrencyLive times out > --- > > Key: HADOOP-15680 > URL: https://issues.apache.org/jira/browse/HADOOP-15680 > Project: Hadoop Common > Issue Type: Bug >Reporter: Andras Bokor >Assignee: Andras Bokor >Priority: Major > Fix For: 3.1.2 > > Attachments: HADOOP-15680.001.patch, HADOOP-15680.002.patch > > > When I am running tests locally ITestNativeAzureFileSystemConcurrencyLive > sometimes times out. > I would like to increase the timeout to avoid unnecessary noise. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15705) Typo in the definition of "stable" in the interface classification
[ https://issues.apache.org/jira/browse/HADOOP-15705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598143#comment-16598143 ] Hudson commented on HADOOP-15705: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14855 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14855/]) HADOOP-15705. Typo in the definition of "stable" in the interface (templedf: rev d53a10b0a552155de700e396fd7f450a4c5f9c22) * (edit) hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md > Typo in the definition of "stable" in the interface classification > -- > > Key: HADOOP-15705 > URL: https://issues.apache.org/jira/browse/HADOOP-15705 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15705.001.patch > > > "Compatible changes allowed: maintenance (x.Y.0)" > should be > "Compatible changes allowed: maintenance (x.y.Z)" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15698) KMS log4j is not initialized properly at startup
[ https://issues.apache.org/jira/browse/HADOOP-15698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598145#comment-16598145 ] Hudson commented on HADOOP-15698: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14855 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14855/]) HADOOP-15698. KMS log4j is not initialized properly at startup. (xiao: rev 781437c219dc3422797a32dc7ba72cd4f5ee38e2) * (edit) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java * (edit) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSConfiguration.java * (edit) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebServer.java > KMS log4j is not initialized properly at startup > > > Key: HADOOP-15698 > URL: https://issues.apache.org/jira/browse/HADOOP-15698 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.1.0 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15698.001.patch > > > During KMs startup, log4j logs don't show up resulting in important logs > getting omitted. This happens because log4 initialisation only happens in > KMSWebApp#contextInitialized and logs written before that don't show up. > For example the following log never shows up: > [https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java#L197-L199] > Another example is that the KMS startup message never shows up in the kms > logs. > Note that this works in the unit tests, because MiniKMS sets the log4j system > property. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15667) FileSystemMultipartUploader should verify that UploadHandle has non-0 length
[ https://issues.apache.org/jira/browse/HADOOP-15667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598146#comment-16598146 ] Hudson commented on HADOOP-15667: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14855 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14855/]) HADOOP-15667. FileSystemMultipartUploader should verify that (stevel: rev 2e6c1109dcdeedb59a3345047e9201271c9a0b27) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/MultipartUploader.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystemMultipartUploader.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractMultipartUploaderTest.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AMultipartUploader.java > FileSystemMultipartUploader should verify that UploadHandle has non-0 length > > > Key: HADOOP-15667 > URL: https://issues.apache.org/jira/browse/HADOOP-15667 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Ewan Higgs >Assignee: Ewan Higgs >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15667.001.patch > > > The S3AMultipartUploader has a good check on the length of the UploadHandle. > This should be moved to MultipartUploader, made protected, and called in the > various implementations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15706) Typo in compatibility doc: SHOUD -> SHOULD
[
https://issues.apache.org/jira/browse/HADOOP-15706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598149#comment-16598149
]
Hudson commented on HADOOP-15706:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14855 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14855/])
HADOOP-15706. Typo in compatibility doc: SHOUD -> SHOULD (Contributed by
(templedf: rev f2c2a68ec208f640e778fc41f95f0284fcc44729)
* (edit) hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md
> Typo in compatibility doc: SHOUD -> SHOULD
> --
>
> Key: HADOOP-15706
> URL: https://issues.apache.org/jira/browse/HADOOP-15706
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Daniel Templeton
>Assignee: Laszlo Kollar
>Priority: Trivial
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HADOOP-15706.001.patch
>
>
> {quote}% grep SHOUD
> ./hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md
> Apache Hadoop revisions SHOUD retain binary compatability such that
> end-user{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14833) Remove s3a user:secret authentication
[ https://issues.apache.org/jira/browse/HADOOP-14833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595664#comment-16595664 ] Mingliang Liu edited comment on HADOOP-14833 at 8/31/18 1:45 AM: - Thanks Steve. I'm +1 on the proposal, and will finish the review this week (EDIT: hopefully). Don't get blocked by me if there is any binding +1. was (Author: liuml07): Thanks Steve. I'm +1 on the proposal, and will finish the review this week. Don't get blocked by me if there is any binding +1. > Remove s3a user:secret authentication > - > > Key: HADOOP-14833 > URL: https://issues.apache.org/jira/browse/HADOOP-14833 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14833-001.patch > > > Remove the s3a://user:secret@host auth mechanism from S3a. > As well as being insecure, it causes problems with S3Guard's URI matching > code. > Proposed: cull it utterly. We've been telling people to stop using it since > HADOOP-3733 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[ https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15707: Description: Hadoop has a few services with HA setups and it is common to set them behind Load Balancers. We should add a way for the Load Balancers to understand what should be the UI to show. For example, the standby RM just redirects the requests to the active RM. However, if both RMs are behind a Load Balancer the IP might not be reachable. Most Load balancers have probes to check if a server reports HTTP code 200: * [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] * [AWS|https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-healthchecks.html] Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to report if they are active. was: Hadoop has a few services with HA setups and it is common to set them behind Load Balancers. We should add a way for the Load Balancers to understand what should be the UI to show. For example, the standby RM just redirects the requests to the active RM. However, if both RMs are behind a Load Balancer the IP might not be reachable. Most Load balancers have probes to check if a server reports HTTP code 200: * [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] * [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to report if they are active. > Add IsActiveServlet to be used for Load Balancers > - > > Key: HADOOP-15707 > URL: https://issues.apache.org/jira/browse/HADOOP-15707 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15707.000.patch, HADOOP-15707.001.patch, > HADOOP-15707.002.patch, HADOOP-15707.003.patch > > > Hadoop has a few services with HA setups and it is common to set them behind > Load Balancers. > We should add a way for the Load Balancers to understand what should be the > UI to show. > For example, the standby RM just redirects the requests to the active RM. > However, if both RMs are behind a Load Balancer the IP might not be reachable. > Most Load balancers have probes to check if a server reports HTTP code 200: > * > [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > * > [AWS|https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-healthchecks.html] > Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to > report if they are active. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[ https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598109#comment-16598109 ] Lukas Majercak commented on HADOOP-15707: - Updated documentation in HA pages for this in patch003. > Add IsActiveServlet to be used for Load Balancers > - > > Key: HADOOP-15707 > URL: https://issues.apache.org/jira/browse/HADOOP-15707 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15707.000.patch, HADOOP-15707.001.patch, > HADOOP-15707.002.patch, HADOOP-15707.003.patch > > > Hadoop has a few services with HA setups and it is common to set them behind > Load Balancers. > We should add a way for the Load Balancers to understand what should be the > UI to show. > For example, the standby RM just redirects the requests to the active RM. > However, if both RMs are behind a Load Balancer the IP might not be reachable. > Most Load balancers have probes to check if a server reports HTTP code 200: > * > [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > * > [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to > report if they are active. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[ https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15707: Attachment: HADOOP-15707.003.patch > Add IsActiveServlet to be used for Load Balancers > - > > Key: HADOOP-15707 > URL: https://issues.apache.org/jira/browse/HADOOP-15707 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15707.000.patch, HADOOP-15707.001.patch, > HADOOP-15707.002.patch, HADOOP-15707.003.patch > > > Hadoop has a few services with HA setups and it is common to set them behind > Load Balancers. > We should add a way for the Load Balancers to understand what should be the > UI to show. > For example, the standby RM just redirects the requests to the active RM. > However, if both RMs are behind a Load Balancer the IP might not be reachable. > Most Load balancers have probes to check if a server reports HTTP code 200: > * > [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > * > [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to > report if they are active. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15696) KMS performance regression due to too many open file descriptors after Jetty migration
[ https://issues.apache.org/jira/browse/HADOOP-15696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598091#comment-16598091 ] Misha Dmitriev commented on HADOOP-15696: - [~xiaochen] I agree that any work on connection reusing is more complex and should be done under another jira. Can you elaborate on "the connection object is created in relation to the caller ugi" statement? What's the exact condition for when a connection can be reused or not? > KMS performance regression due to too many open file descriptors after Jetty > migration > -- > > Key: HADOOP-15696 > URL: https://issues.apache.org/jira/browse/HADOOP-15696 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Attachments: HADOOP-15696.001.patch, Screen Shot 2018-08-22 at > 11.36.16 AM.png, Screen Shot 2018-08-22 at 4.26.51 PM.png, Screen Shot > 2018-08-22 at 4.26.51 PM.png, Screen Shot 2018-08-22 at 4.27.02 PM.png, > Screen Shot 2018-08-22 at 4.30.32 PM.png, Screen Shot 2018-08-22 at 4.30.39 > PM.png, Screen Shot 2018-08-24 at 7.08.16 PM.png > > > We recently found KMS performance regressed in Hadoop 3.0, possibly linking > to the migration from Tomcat to Jetty in HADOOP-13597. > Symptoms: > # Hadoop 3.x KMS open file descriptors quickly rises to more than 10 thousand > under stress, sometimes even exceeds 32K, which is the system limit, causing > failures for any access to encryption zones. Our internal testing shows the > openfd number was in the range of a few hundred in Hadoop 2.x, and it > increases by almost 100x in Hadoop 3. > # Hadoop 3.x KMS as much as twice the heap size than in Hadoop 2.x. The same > heap size can go OOM in Hadoop 3.x. Jxray analysis suggests most of them are > temporary byte arrays associated with open SSL connections. > # Due to the heap usage, Hadoop 3.x KMS has more frequent GC activities, and > we observed up to 20% performance reduction due to GC. > A possible solution is to reduce the idle timeout setting in HttpServer2. It > is currently hard-coded 10 seconds. By setting it to 1 second, open fds > dropped from 20 thousand down to 3 thousand in my experiment. > File this jira to invite open discussion for a solution. > Credit: [~mi...@cloudera.com] for the proposed Jetty idle timeout remedy; > [~xiaochen] for digging into this problem. > Screenshots: > CDH5 (Hadoop 2) KMS CPU utilization, resident memory and file descriptor > chart. > !Screen Shot 2018-08-22 at 4.30.39 PM.png! > CDH6 (Hadoop 3) KMS CPU utilization, resident memory and file descriptor > chart. > !Screen Shot 2018-08-22 at 4.30.32 PM.png! > CDH5 (Hadoop 2) GC activities on the KMS process > !Screen Shot 2018-08-22 at 4.26.51 PM.png! > CDH6 (Hadoop 3) GC activities on the KMS process > !Screen Shot 2018-08-22 at 4.27.02 PM.png! > JXray report > !Screen Shot 2018-08-22 at 11.36.16 AM.png! > open fd drops from 20 k down to 3k after the proposed change. > !Screen Shot 2018-08-24 at 7.08.16 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[
https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598029#comment-16598029
]
Íñigo Goiri edited comment on HADOOP-15707 at 8/30/18 11:12 PM:
Thanks [~virajith] for the comments.
We have been using it for a couple weeks with our internal Load Balancers.
Today tested it in Azure VMSS and it seems to do the trick, this is the example
for how to set it for the RM:
{code}
az network lb probe create -g XXX --lb-name YYY -n RMProbe --protocol http
--port 8088 --path /isActive
{code}
was (Author: elgoiri):
Thanks [~virajith] for the comments.
We have been using it for a couple weeks with our internal Load Balancers.
Today we had tested it in Azure VMSS and it seems to do the trick, this is the
example for how to set it for the RM:
{code}
az network lb probe create -g XXX --lb-name YYY -n RMProbe --protocol http
--port 8088 --path /isActive
{code}
> Add IsActiveServlet to be used for Load Balancers
> -
>
> Key: HADOOP-15707
> URL: https://issues.apache.org/jira/browse/HADOOP-15707
> Project: Hadoop Common
> Issue Type: New Feature
> Components: common
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Major
> Attachments: HADOOP-15707.000.patch, HADOOP-15707.001.patch,
> HADOOP-15707.002.patch
>
>
> Hadoop has a few services with HA setups and it is common to set them behind
> Load Balancers.
> We should add a way for the Load Balancers to understand what should be the
> UI to show.
> For example, the standby RM just redirects the requests to the active RM.
> However, if both RMs are behind a Load Balancer the IP might not be reachable.
> Most Load balancers have probes to check if a server reports HTTP code 200:
> *
> [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview]
> *
> [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview]
> Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to
> report if they are active.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[
https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598029#comment-16598029
]
Íñigo Goiri commented on HADOOP-15707:
--
Thanks [~virajith] for the comments.
We have been using it for a couple weeks with our internal Load Balancers.
Today we had tested it in Azure VMSS and it seems to do the trick, this is the
example for how to set it for the RM:
{code}
az network lb probe create -g XXX --lb-name YYY -n RMProbe --protocol http
--port 8088 --path /isActive
{code}
> Add IsActiveServlet to be used for Load Balancers
> -
>
> Key: HADOOP-15707
> URL: https://issues.apache.org/jira/browse/HADOOP-15707
> Project: Hadoop Common
> Issue Type: New Feature
> Components: common
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Major
> Attachments: HADOOP-15707.000.patch, HADOOP-15707.001.patch,
> HADOOP-15707.002.patch
>
>
> Hadoop has a few services with HA setups and it is common to set them behind
> Load Balancers.
> We should add a way for the Load Balancers to understand what should be the
> UI to show.
> For example, the standby RM just redirects the requests to the active RM.
> However, if both RMs are behind a Load Balancer the IP might not be reachable.
> Most Load balancers have probes to check if a server reports HTTP code 200:
> *
> [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview]
> *
> [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview]
> Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to
> report if they are active.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[ https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598019#comment-16598019 ] Virajith Jalaparti commented on HADOOP-15707: - I don't know this part of the code much but the patch itself looks mostly good to me. Has this been tested with Azure/AWS load balancers? If so, would be great to see any info on that. > Add IsActiveServlet to be used for Load Balancers > - > > Key: HADOOP-15707 > URL: https://issues.apache.org/jira/browse/HADOOP-15707 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15707.000.patch, HADOOP-15707.001.patch, > HADOOP-15707.002.patch > > > Hadoop has a few services with HA setups and it is common to set them behind > Load Balancers. > We should add a way for the Load Balancers to understand what should be the > UI to show. > For example, the standby RM just redirects the requests to the active RM. > However, if both RMs are behind a Load Balancer the IP might not be reachable. > Most Load balancers have probes to check if a server reports HTTP code 200: > * > [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > * > [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to > report if they are active. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15694) ABFS: Allow OAuth credentials to not be tied to accounts
[ https://issues.apache.org/jira/browse/HADOOP-15694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597993#comment-16597993 ] Sean Mackrory commented on HADOOP-15694: Attaching a patch that's basically what I want to do. Feedback welcome, but I'm not entirely finished with this so I'm not proposing it for inclusion yet. Integration tests are running now, but so far so good. I also still need to add my own tests and do my own code review for silly mistakes, but I was able to connect to ABFS by just specifying "fs.azure.account.key" without any account context (although that'll be more useful with ABFS), and that's basically what I wanted to achieve. > ABFS: Allow OAuth credentials to not be tied to accounts > > > Key: HADOOP-15694 > URL: https://issues.apache.org/jira/browse/HADOOP-15694 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Attachments: HADOOP-15694.001.patch > > > Now that there's OAuth support, it's possible to have a notion of identity > that's distinct from the account itself. If a cluster is configured via OAuth > with it's own identity, it's likely operators will want to use that identity > regardless of which storage account a job uses. > So OAuth configs right now (and probably others) are looked up with > .. I propose that we add a function for looking up these > configs that returns an account-specific value if it exists, but in the event > it does not will also try to return , if that exists. > I can work on a patch for this if nobody has any objections. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15694) ABFS: Allow OAuth credentials to not be tied to accounts
[ https://issues.apache.org/jira/browse/HADOOP-15694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HADOOP-15694: --- Attachment: HADOOP-15694.001.patch > ABFS: Allow OAuth credentials to not be tied to accounts > > > Key: HADOOP-15694 > URL: https://issues.apache.org/jira/browse/HADOOP-15694 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Attachments: HADOOP-15694.001.patch > > > Now that there's OAuth support, it's possible to have a notion of identity > that's distinct from the account itself. If a cluster is configured via OAuth > with it's own identity, it's likely operators will want to use that identity > regardless of which storage account a job uses. > So OAuth configs right now (and probably others) are looked up with > .. I propose that we add a function for looking up these > configs that returns an account-specific value if it exists, but in the event > it does not will also try to return , if that exists. > I can work on a patch for this if nobody has any objections. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[ https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597972#comment-16597972 ] Lukas Majercak edited comment on HADOOP-15707 at 8/30/18 10:07 PM: --- Add patch002 for setting up IsRouterActiveServlet in RouterHttpServer was (Author: lukmajercak): Add -HADOOP-1570+7+-.002.patch for setting up IsRouterActiveServlet in RouterHttpServer > Add IsActiveServlet to be used for Load Balancers > - > > Key: HADOOP-15707 > URL: https://issues.apache.org/jira/browse/HADOOP-15707 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15707.000.patch, HADOOP-15707.001.patch, > HADOOP-15707.002.patch > > > Hadoop has a few services with HA setups and it is common to set them behind > Load Balancers. > We should add a way for the Load Balancers to understand what should be the > UI to show. > For example, the standby RM just redirects the requests to the active RM. > However, if both RMs are behind a Load Balancer the IP might not be reachable. > Most Load balancers have probes to check if a server reports HTTP code 200: > * > [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > * > [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to > report if they are active. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[ https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597972#comment-16597972 ] Lukas Majercak commented on HADOOP-15707: - Add HADOOP-15705.002.patch for setting up IsRouterActiveServlet in RouterHttpServer > Add IsActiveServlet to be used for Load Balancers > - > > Key: HADOOP-15707 > URL: https://issues.apache.org/jira/browse/HADOOP-15707 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15707.000.patch, HADOOP-15707.001.patch, > HADOOP-15707.002.patch > > > Hadoop has a few services with HA setups and it is common to set them behind > Load Balancers. > We should add a way for the Load Balancers to understand what should be the > UI to show. > For example, the standby RM just redirects the requests to the active RM. > However, if both RMs are behind a Load Balancer the IP might not be reachable. > Most Load balancers have probes to check if a server reports HTTP code 200: > * > [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > * > [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to > report if they are active. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[ https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597972#comment-16597972 ] Lukas Majercak edited comment on HADOOP-15707 at 8/30/18 10:06 PM: --- Add -HADOOP-1570+7+-.002.patch for setting up IsRouterActiveServlet in RouterHttpServer was (Author: lukmajercak): Add HADOOP-15705.002.patch for setting up IsRouterActiveServlet in RouterHttpServer > Add IsActiveServlet to be used for Load Balancers > - > > Key: HADOOP-15707 > URL: https://issues.apache.org/jira/browse/HADOOP-15707 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15707.000.patch, HADOOP-15707.001.patch, > HADOOP-15707.002.patch > > > Hadoop has a few services with HA setups and it is common to set them behind > Load Balancers. > We should add a way for the Load Balancers to understand what should be the > UI to show. > For example, the standby RM just redirects the requests to the active RM. > However, if both RMs are behind a Load Balancer the IP might not be reachable. > Most Load balancers have probes to check if a server reports HTTP code 200: > * > [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > * > [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to > report if they are active. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[ https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15707: Attachment: HADOOP-15707.002.patch > Add IsActiveServlet to be used for Load Balancers > - > > Key: HADOOP-15707 > URL: https://issues.apache.org/jira/browse/HADOOP-15707 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15707.000.patch, HADOOP-15707.001.patch, > HADOOP-15707.002.patch > > > Hadoop has a few services with HA setups and it is common to set them behind > Load Balancers. > We should add a way for the Load Balancers to understand what should be the > UI to show. > For example, the standby RM just redirects the requests to the active RM. > However, if both RMs are behind a Load Balancer the IP might not be reachable. > Most Load balancers have probes to check if a server reports HTTP code 200: > * > [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > * > [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to > report if they are active. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15703) ABFS - Implement client-side throttling
[ https://issues.apache.org/jira/browse/HADOOP-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597969#comment-16597969 ] Da Zhou commented on HADOOP-15703: -- AbfsClientThrottlingIntercept: L76, please remove "java.net.", it doesn't need to be full qualified name. L99: ABFS Gen2 doesn't depend on Azure Storage SDK, please update the comments. > ABFS - Implement client-side throttling > > > Key: HADOOP-15703 > URL: https://issues.apache.org/jira/browse/HADOOP-15703 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sneha Varma >Assignee: Sneha Varma >Priority: Major > Attachments: HADOOP-15703-HADOOP-15407-001.patch > > > Big data workloads frequently exceed the AzureBlobFS max ingress and egress > limits > (https://docs.microsoft.com/en-us/azure/storage/common/storage-scalability-targets). > For example, the max ingress limit for a GRS account in the United States is > currently 10 Gbps. When the limit is exceeded, the AzureBlobFS service fails > a percentage of incoming requests, and this causes the client to initiate the > retry policy. The retry policy delays requests by sleeping, but the sleep > duration is independent of the client throughput and account limit. This > results in low throughput, due to the high number of failed requests and > thrashing causes by the retry policy. > To fix this, we introduce a client-side throttle which minimizes failed > requests and maximizes throughput. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[ https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597966#comment-16597966 ] Lukas Majercak commented on HADOOP-15707: - Added patch 001 with updated comments. I'll add the documentation in HA pages in a bit. > Add IsActiveServlet to be used for Load Balancers > - > > Key: HADOOP-15707 > URL: https://issues.apache.org/jira/browse/HADOOP-15707 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15707.000.patch, HADOOP-15707.001.patch > > > Hadoop has a few services with HA setups and it is common to set them behind > Load Balancers. > We should add a way for the Load Balancers to understand what should be the > UI to show. > For example, the standby RM just redirects the requests to the active RM. > However, if both RMs are behind a Load Balancer the IP might not be reachable. > Most Load balancers have probes to check if a server reports HTTP code 200: > * > [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > * > [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to > report if they are active. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[ https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15707: Attachment: HADOOP-15707.001.patch > Add IsActiveServlet to be used for Load Balancers > - > > Key: HADOOP-15707 > URL: https://issues.apache.org/jira/browse/HADOOP-15707 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15707.000.patch, HADOOP-15707.001.patch > > > Hadoop has a few services with HA setups and it is common to set them behind > Load Balancers. > We should add a way for the Load Balancers to understand what should be the > UI to show. > For example, the standby RM just redirects the requests to the active RM. > However, if both RMs are behind a Load Balancer the IP might not be reachable. > Most Load balancers have probes to check if a server reports HTTP code 200: > * > [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > * > [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to > report if they are active. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[ https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HADOOP-15707: - Description: Hadoop has a few services with HA setups and it is common to set them behind Load Balancers. We should add a way for the Load Balancers to understand what should be the UI to show. For example, the standby RM just redirects the requests to the active RM. However, if both RMs are behind a Load Balancer the IP might not be reachable. Most Load balancers have probes to check if a server reports HTTP code 200: * [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] * [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to report if they are active. was: Hadoop has a few services with HA setups and it is common to set them behind Load Balancers. We should add a way for the Load Balancers to understand what should be the UI to show. For example, the standby RM just redirects the requests to the active RM. However, if both RMs are behind a Load Balancer the IP might not be reachable. Most Load balancers have probes to check if a server reports HTTP code 200: * [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] * [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to report if they are active. > Add IsActiveServlet to be used for Load Balancers > - > > Key: HADOOP-15707 > URL: https://issues.apache.org/jira/browse/HADOOP-15707 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15707.000.patch > > > Hadoop has a few services with HA setups and it is common to set them behind > Load Balancers. > We should add a way for the Load Balancers to understand what should be the > UI to show. > For example, the standby RM just redirects the requests to the active RM. > However, if both RMs are behind a Load Balancer the IP might not be reachable. > Most Load balancers have probes to check if a server reports HTTP code 200: > * > [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > * > [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to > report if they are active. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[ https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HADOOP-15707: - Description: Hadoop has a few services with HA setups and it is common to set them behind Load Balancers. We should add a way for the Load Balancers to understand what should be the UI to show. For example, the standby RM just redirects the requests to the active RM. However, if both RMs are behind a Load Balancer the IP might not be reachable. Most Load balancers have probes to check if a server reports HTTP code 200: * [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] * [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to report if they are active. > Add IsActiveServlet to be used for Load Balancers > - > > Key: HADOOP-15707 > URL: https://issues.apache.org/jira/browse/HADOOP-15707 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15707.000.patch > > > Hadoop has a few services with HA setups and it is common to set them behind > Load Balancers. > We should add a way for the Load Balancers to understand what should be the > UI to show. > For example, the standby RM just redirects the requests to the active RM. > However, if both RMs are behind a Load Balancer the IP might not be reachable. > Most Load balancers have probes to check if a server reports HTTP code 200: > * > [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > * > [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to > report if they are active. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[ https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597942#comment-16597942 ] Íñigo Goiri commented on HADOOP-15707: -- Thanks [~lukmajercak] for working on this. Right now, [^HADOOP-15707.000.patch] touches Commons, YARN (RM), and HDFS (NN and Router). I think this is small enough that it could go in this single patch. We may want to add some documentation maybe in the HA pages. I'll update the description with the use cases. > Add IsActiveServlet to be used for Load Balancers > - > > Key: HADOOP-15707 > URL: https://issues.apache.org/jira/browse/HADOOP-15707 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15707.000.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[ https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15707: Attachment: HADOOP-15707.000.patch > Add IsActiveServlet to be used for Load Balancers > - > > Key: HADOOP-15707 > URL: https://issues.apache.org/jira/browse/HADOOP-15707 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15707.000.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
Lukas Majercak created HADOOP-15707: --- Summary: Add IsActiveServlet to be used for Load Balancers Key: HADOOP-15707 URL: https://issues.apache.org/jira/browse/HADOOP-15707 Project: Hadoop Common Issue Type: New Feature Components: common Reporter: Lukas Majercak Assignee: Lukas Majercak -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10219) ipc.Client.setupIOstreams() needs to check for ClientCache.stopClient requested shutdowns
[
https://issues.apache.org/jira/browse/HADOOP-10219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597910#comment-16597910
]
Lukas Majercak commented on HADOOP-10219:
-
Kindly ping.
> ipc.Client.setupIOstreams() needs to check for ClientCache.stopClient
> requested shutdowns
> --
>
> Key: HADOOP-10219
> URL: https://issues.apache.org/jira/browse/HADOOP-10219
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.2.0, 2.6.0
>Reporter: Steve Loughran
>Assignee: Kihwal Lee
>Priority: Major
> Attachments: HADOOP-10219.patch, HADOOP-10219.v1.patch,
> HADOOP-10219.v2.patch, HADOOP-10219.v3.patch, HADOOP-10219.v4.patch
>
>
> When {{ClientCache.stopClient()}} is called to stop the IPC client, if the
> client
> is blocked spinning due to a connectivity problem, it does not exit until
> the policy has timed out -so the stopClient() operation can hang for an
> extended period of time.
> This can surface in the shutdown hook of FileSystem.cache.closeAll()
> Also, Client.stop() is for used in NN switch from Standby to Active, and can
> therefore have very bad consequences and cause downtime.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14734) add option to tag DDB table(s) created
[ https://issues.apache.org/jira/browse/HADOOP-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-14734: --- Assignee: Gabor Bota (was: Abraham Fine) > add option to tag DDB table(s) created > -- > > Key: HADOOP-14734 > URL: https://issues.apache.org/jira/browse/HADOOP-14734 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-14734-001.patch, HADOOP-14734-002.patch, > HADOOP-14734-003.patch > > > Many organisations have a "no untagged" resource policy; s3guard runs into > this when a table is created untagged. If there's a strict "delete untagged > resources" policy, the tables will go without warning. > Proposed: we add an option which can be used to declare the tags for a table > when created, use it in creation. No need to worry about updating/viewing > tags, as the AWS console can do that -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15426) Make S3guard client resilient to DDB throttle events and network failures
[
https://issues.apache.org/jira/browse/HADOOP-15426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15426:
Status: Patch Available (was: Open)
patch 008: patch 007 with checkstyles corrected.
looks like we've been slowly adding legit checkstyle errors to the code; needs
a fixup patch which I'm not going to do here, as it will complicate cherry
picking this back
> Make S3guard client resilient to DDB throttle events and network failures
> -
>
> Key: HADOOP-15426
> URL: https://issues.apache.org/jira/browse/HADOOP-15426
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15426-001.patch, HADOOP-15426-002.patch,
> HADOOP-15426-003.patch, HADOOP-15426-004.patch, HADOOP-15426-005.patch,
> HADOOP-15426-006.patch, HADOOP-15426-007.patch, HADOOP-15426-008.patch,
> Screen Shot 2018-07-24 at 15.16.46.png, Screen Shot 2018-07-25 at
> 16.22.10.png, Screen Shot 2018-07-25 at 16.28.53.png, Screen Shot 2018-07-27
> at 14.07.38.png,
> org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStoreScale-output.txt
>
>
> managed to create on a parallel test run
> {code}
> org.apache.hadoop.fs.s3a.AWSServiceThrottledException: delete on
> s3a://hwdev-steve-ireland-new/fork-0005/test/existing-dir/existing-file:
> com.amazonaws.services.dynamodbv2.model.ProvisionedThroughputExceededException:
> The level of configured provisioned throughput for the table was exceeded.
> Consider increasing your provisioning level with the UpdateTable API.
> (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
> ProvisionedThroughputExceededException; Request ID:
> RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG): The level of
> configured provisioned throughput for the table was exceeded. Consider
> increasing your provisioning level with the UpdateTable API. (Service:
> AmazonDynamoDBv2; Status Code: 400; Error Code:
> ProvisionedThroughputExceededException; Request ID:
> RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG)
> at
> {code}
> We should be able to handle this. 400 "bad things happened" error though, not
> the 503 from S3.
> h3. We need a retry handler for DDB throttle operations
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15426) Make S3guard client resilient to DDB throttle events and network failures
[
https://issues.apache.org/jira/browse/HADOOP-15426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15426:
Attachment: HADOOP-15426-008.patch
> Make S3guard client resilient to DDB throttle events and network failures
> -
>
> Key: HADOOP-15426
> URL: https://issues.apache.org/jira/browse/HADOOP-15426
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15426-001.patch, HADOOP-15426-002.patch,
> HADOOP-15426-003.patch, HADOOP-15426-004.patch, HADOOP-15426-005.patch,
> HADOOP-15426-006.patch, HADOOP-15426-007.patch, HADOOP-15426-008.patch,
> Screen Shot 2018-07-24 at 15.16.46.png, Screen Shot 2018-07-25 at
> 16.22.10.png, Screen Shot 2018-07-25 at 16.28.53.png, Screen Shot 2018-07-27
> at 14.07.38.png,
> org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStoreScale-output.txt
>
>
> managed to create on a parallel test run
> {code}
> org.apache.hadoop.fs.s3a.AWSServiceThrottledException: delete on
> s3a://hwdev-steve-ireland-new/fork-0005/test/existing-dir/existing-file:
> com.amazonaws.services.dynamodbv2.model.ProvisionedThroughputExceededException:
> The level of configured provisioned throughput for the table was exceeded.
> Consider increasing your provisioning level with the UpdateTable API.
> (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
> ProvisionedThroughputExceededException; Request ID:
> RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG): The level of
> configured provisioned throughput for the table was exceeded. Consider
> increasing your provisioning level with the UpdateTable API. (Service:
> AmazonDynamoDBv2; Status Code: 400; Error Code:
> ProvisionedThroughputExceededException; Request ID:
> RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG)
> at
> {code}
> We should be able to handle this. 400 "bad things happened" error though, not
> the 503 from S3.
> h3. We need a retry handler for DDB throttle operations
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15426) Make S3guard client resilient to DDB throttle events and network failures
[
https://issues.apache.org/jira/browse/HADOOP-15426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15426:
Status: Open (was: Patch Available)
> Make S3guard client resilient to DDB throttle events and network failures
> -
>
> Key: HADOOP-15426
> URL: https://issues.apache.org/jira/browse/HADOOP-15426
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15426-001.patch, HADOOP-15426-002.patch,
> HADOOP-15426-003.patch, HADOOP-15426-004.patch, HADOOP-15426-005.patch,
> HADOOP-15426-006.patch, HADOOP-15426-007.patch, Screen Shot 2018-07-24 at
> 15.16.46.png, Screen Shot 2018-07-25 at 16.22.10.png, Screen Shot 2018-07-25
> at 16.28.53.png, Screen Shot 2018-07-27 at 14.07.38.png,
> org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStoreScale-output.txt
>
>
> managed to create on a parallel test run
> {code}
> org.apache.hadoop.fs.s3a.AWSServiceThrottledException: delete on
> s3a://hwdev-steve-ireland-new/fork-0005/test/existing-dir/existing-file:
> com.amazonaws.services.dynamodbv2.model.ProvisionedThroughputExceededException:
> The level of configured provisioned throughput for the table was exceeded.
> Consider increasing your provisioning level with the UpdateTable API.
> (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
> ProvisionedThroughputExceededException; Request ID:
> RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG): The level of
> configured provisioned throughput for the table was exceeded. Consider
> increasing your provisioning level with the UpdateTable API. (Service:
> AmazonDynamoDBv2; Status Code: 400; Error Code:
> ProvisionedThroughputExceededException; Request ID:
> RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG)
> at
> {code}
> We should be able to handle this. 400 "bad things happened" error though, not
> the 503 from S3.
> h3. We need a retry handler for DDB throttle operations
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15657) Registering MutableQuantiles via Metric annotation
[
https://issues.apache.org/jira/browse/HADOOP-15657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597625#comment-16597625
]
Vrushali C commented on HADOOP-15657:
-
+1 LGTM
thanks for the patch [~Sushil-K-S]!
I will commit this later today.
> Registering MutableQuantiles via Metric annotation
> --
>
> Key: HADOOP-15657
> URL: https://issues.apache.org/jira/browse/HADOOP-15657
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Reporter: Sushil Ks
>Priority: Major
> Attachments: HADOOP-15657.001.patch
>
>
> Currently when creating new metrics we use @Metric annotation for registering
> the MutableMetric i.e
> {code:java}
> @Metric
> private MutableInt foobarMetricCount
> {code}
> However, there's no support for registering MutableQuantiles via Metric
> annotation, hence creating this Jira to register MutableQuantiles via Metric
> annotation.
> Example:
>
> {code:java}
> @Metric(about = "async PUT entities latency", valueName = "latency", interval
> = 10)
> private MutableQuantiles foobarAsyncLatency;
>
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15706) Typo in compatibility doc: SHOUD -> SHOULD
[
https://issues.apache.org/jira/browse/HADOOP-15706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated HADOOP-15706:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.2
3.0.4
3.2.0
Status: Resolved (was: Patch Available)
Thanks for the patch! Committed to trunk, branch-3.1, and branch-3.0.
> Typo in compatibility doc: SHOUD -> SHOULD
> --
>
> Key: HADOOP-15706
> URL: https://issues.apache.org/jira/browse/HADOOP-15706
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Daniel Templeton
>Assignee: Laszlo Kollar
>Priority: Trivial
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HADOOP-15706.001.patch
>
>
> {quote}% grep SHOUD
> ./hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md
> Apache Hadoop revisions SHOUD retain binary compatability such that
> end-user{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15706) Typo in compatibility doc: SHOUD -> SHOULD
[
https://issues.apache.org/jira/browse/HADOOP-15706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597615#comment-16597615
]
Daniel Templeton commented on HADOOP-15706:
---
+1, thanks for the patch.
> Typo in compatibility doc: SHOUD -> SHOULD
> --
>
> Key: HADOOP-15706
> URL: https://issues.apache.org/jira/browse/HADOOP-15706
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Daniel Templeton
>Assignee: Laszlo Kollar
>Priority: Trivial
> Attachments: HADOOP-15706.001.patch
>
>
> {quote}% grep SHOUD
> ./hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md
> Apache Hadoop revisions SHOUD retain binary compatability such that
> end-user{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15706) Typo in compatibility doc: SHOUD -> SHOULD
[
https://issues.apache.org/jira/browse/HADOOP-15706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Kollar updated HADOOP-15706:
---
Status: Patch Available (was: Open)
> Typo in compatibility doc: SHOUD -> SHOULD
> --
>
> Key: HADOOP-15706
> URL: https://issues.apache.org/jira/browse/HADOOP-15706
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Daniel Templeton
>Assignee: Laszlo Kollar
>Priority: Trivial
> Attachments: HADOOP-15706.001.patch
>
>
> {quote}% grep SHOUD
> ./hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md
> Apache Hadoop revisions SHOUD retain binary compatability such that
> end-user{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15706) Typo in compatibility doc: SHOUD -> SHOULD
[
https://issues.apache.org/jira/browse/HADOOP-15706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Kollar updated HADOOP-15706:
---
Attachment: HADOOP-15706.001.patch
> Typo in compatibility doc: SHOUD -> SHOULD
> --
>
> Key: HADOOP-15706
> URL: https://issues.apache.org/jira/browse/HADOOP-15706
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Daniel Templeton
>Assignee: Laszlo Kollar
>Priority: Trivial
> Attachments: HADOOP-15706.001.patch
>
>
> {quote}% grep SHOUD
> ./hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md
> Apache Hadoop revisions SHOUD retain binary compatability such that
> end-user{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15706) Typo in compatibility doc: SHOUD -> SHOULD
[
https://issues.apache.org/jira/browse/HADOOP-15706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton reassigned HADOOP-15706:
-
Assignee: Laszlo Kollar (was: Daniel Templeton)
> Typo in compatibility doc: SHOUD -> SHOULD
> --
>
> Key: HADOOP-15706
> URL: https://issues.apache.org/jira/browse/HADOOP-15706
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Daniel Templeton
>Assignee: Laszlo Kollar
>Priority: Trivial
>
> {quote}% grep SHOUD
> ./hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md
> Apache Hadoop revisions SHOUD retain binary compatability such that
> end-user{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15639) Classloader issue with hadoop cmd -libjars option
[
https://issues.apache.org/jira/browse/HADOOP-15639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597575#comment-16597575
]
Qinghui Xu commented on HADOOP-15639:
-
Here is a patch to fix it:
https://github.com/criteo-forks/hadoop-common/pull/87/commits/da6fdfa53410906068383550b6214b7a81344083
> Classloader issue with hadoop cmd -libjars option
> --
>
> Key: HADOOP-15639
> URL: https://issues.apache.org/jira/browse/HADOOP-15639
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Qinghui Xu
>Priority: Major
>
> '-libjars' option gives the possibility to add extra libraries in a hadoop
> command by providing a custom URLClassLoader to fetch jars regarding to the
> option.
> The classloader is provided to both hadoop Configuration and the main
> thread's context classloader, so that when calling Configuration#getClass or
> using Class.forName the class can be loaded by the URLClassLoader.
> But two instances of URLClassLoader are provided to Configuration and to
> thread context classloader respectively, which makes a same class possible to
> be loaded twice by two classloaders. And class loaded by different
> classloaders is considered different.
> So the following assertion will fail:
> {code:java}
> // CustomFileSystem is provided by -libjars option
> Class clazz1 = conf.getClass("CustomFileSystem");
> Class clazz2 = Class.forName("CustomFileSystem");
> // The two classes are different
> assert clazz1 == clazz2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15107) Stabilize/tune S3A committers; review correctness & docs
[ https://issues.apache.org/jira/browse/HADOOP-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15107: Resolution: Fixed Fix Version/s: 3.1.2 Status: Resolved (was: Patch Available) > Stabilize/tune S3A committers; review correctness & docs > > > Key: HADOOP-15107 > URL: https://issues.apache.org/jira/browse/HADOOP-15107 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Fix For: 3.1.2 > > Attachments: HADOOP-15107-001.patch, HADOOP-15107-002.patch, > HADOOP-15107-003.patch, HADOOP-15107-004.patch > > > I'm writing about the paper on the committers, one which, being a proper > paper, requires me to show the committers work. > # define the requirements of a "Correct" committed job (this applies to the > FileOutputCommitter too) > # show that the Staging committer meets these requirements (most of this is > implicit in that it uses the V1 FileOutputCommitter to marshall .pendingset > lists from committed tasks to the final destination, where they are read and > committed. > # Show the magic committer also works. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15107) Stabilize/tune S3A committers; review correctness & docs
[
https://issues.apache.org/jira/browse/HADOOP-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597511#comment-16597511
]
Steve Loughran commented on HADOOP-15107:
-
Thanks, committed to 3.1.x & trunk!
bq. New commit attempts always get the same attempt id +1? (I don't know how
those are allocated)
its how they know to recover from the previous attempt. Yarn App ID is used to
guarantee uniqueness over apps. Spark always starts off with 0 as its app
Id/attempt ID, so >1 query from different spark instances can clash.
bq. The mergePathsV1 seems pretty straightforward. Not sure why the actual code
is so complicated.
unplanned evolution is my guess, possibly with a goal of not breaking any
explict subclasses of the FileOutputCommitter. I didn't try that for the new
commit stuff.
v2 resilience?
It is broken in that nothing can handle a task which fails during commit: its
(non-atomic) state is unknown. Neither MapReduce nor Spark are aware
of/resilient to this issue. There's also the partitioning failure mode: task
doesn't fail during commit, it merely hangs for a while (GC?) then completes
its commit when it resumes, without noticing that it's been superceded by a
second task attempt, or indeed, that the entire job has now completed. Oops.
Bear in mind though that outside object stores with slow renames the
probability of a failure during task commit is likely to low.
Really committers should expose their semantics here & MR & Spark can handle
this failure condition.
V1 doesn't have this problem as the task commit is atomic; job commit is not,
but as {{isCommitJobRepeatable()}} returns false for that, MR AM restart knows
to give up then (something is saved to HDFS indicate in-job-commit). Spark
doesn't restart failed AM/driver, so it's moot there.
S3A committers
* Staging: relies on V1 semantics in cluster HDFS
* Magic. Task commit writes {{PendingSet}} of all files to commit to task in an
atomic PUT; task commit is therefore also atomic. After a job completes we
purge all pending uploads under $dest, so any failed tasks' output is deleted.
> Stabilize/tune S3A committers; review correctness & docs
>
>
> Key: HADOOP-15107
> URL: https://issues.apache.org/jira/browse/HADOOP-15107
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Fix For: 3.1.2
>
> Attachments: HADOOP-15107-001.patch, HADOOP-15107-002.patch,
> HADOOP-15107-003.patch, HADOOP-15107-004.patch
>
>
> I'm writing about the paper on the committers, one which, being a proper
> paper, requires me to show the committers work.
> # define the requirements of a "Correct" committed job (this applies to the
> FileOutputCommitter too)
> # show that the Staging committer meets these requirements (most of this is
> implicit in that it uses the V1 FileOutputCommitter to marshall .pendingset
> lists from committed tasks to the final destination, where they are read and
> committed.
> # Show the magic committer also works.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15628) S3A Filesystem does not check return from AmazonS3Client deleteObjects
[
https://issues.apache.org/jira/browse/HADOOP-15628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597497#comment-16597497
]
Steve Loughran commented on HADOOP-15628:
-
we are retrying for network failures but not for failure of partial delete
except when an exception is raised.
But in the AWS SDK, I see the check and uprate to an exception
{Code}
DeleteObjectsResponse response = invoke(request, responseHandler,
deleteObjectsRequest.getBucketName(), null);
if ( !response.getErrors().isEmpty() ) {
..
MultiObjectDeleteException ex = new MultiObjectDeleteException(
response.getErrors(),
response.getDeletedObjects());
...
throw ex;
}
{Code}
it's relying on the errors list being non-empty. So if your store is failing
with some other error code rather than returning a list of rejected files, we
aren't picking it up
> S3A Filesystem does not check return from AmazonS3Client deleteObjects
> --
>
> Key: HADOOP-15628
> URL: https://issues.apache.org/jira/browse/HADOOP-15628
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.1, 2.8.4, 3.1.1, 3.0.3
> Environment: Hadoop 3.0.2 / Hadoop 2.8.3
> Hive 2.3.2 / Hive 2.3.3 / Hive 3.0.0
> Non-AWS S3 implementation
>Reporter: Steve Jacobs
>Priority: Minor
>
> Deletes in S3A that use the Multi-Delete functionality in the Amazon S3 api
> do not check to see if all objects have been succesfully delete. In the event
> of a failure, the api will still return a 200 OK (which isn't checked
> currently):
> [Delete Code from Hadoop
> 2.8|https://github.com/apache/hadoop/blob/a0da1ec01051108b77f86799dd5e97563b2a3962/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L574]
>
> {code:java}
> if (keysToDelete.size() == MAX_ENTRIES_TO_DELETE) {
> DeleteObjectsRequest deleteRequest =
> new DeleteObjectsRequest(bucket).withKeys(keysToDelete);
> s3.deleteObjects(deleteRequest);
> statistics.incrementWriteOps(1);
> keysToDelete.clear();
> }
> {code}
> This should be converted to use the DeleteObjectsResult class from the
> S3Client:
> [Amazon Code
> Example|https://docs.aws.amazon.com/AmazonS3/latest/dev/DeletingMultipleObjectsUsingJava.htm]
> {code:java}
> // Verify that the objects were deleted successfully.
> DeleteObjectsResult delObjRes =
> s3Client.deleteObjects(multiObjectDeleteRequest); int successfulDeletes =
> delObjRes.getDeletedObjects().size();
> System.out.println(successfulDeletes + " objects successfully deleted.");
> {code}
> Bucket policies can be misconfigured, and deletes will fail without warning
> by S3A clients.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14630) Contract Tests to verify create, mkdirs and rename under a file is forbidden
[
https://issues.apache.org/jira/browse/HADOOP-14630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597491#comment-16597491
]
Steve Loughran commented on HADOOP-14630:
-
I'm not sure this is ready to go in; because of the
{{TestLocalFSContractRename}} failure. I was just rebasing it to trunk to see
if it still builds before going near it again
> Contract Tests to verify create, mkdirs and rename under a file is forbidden
>
>
> Key: HADOOP-14630
> URL: https://issues.apache.org/jira/browse/HADOOP-14630
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/azure, fs/s3, fs/swift
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14630-001.patch, HADOOP-14630-002.patch,
> HADOOP-14630-003.patch, HADOOP-14630-004.patch
>
>
> Object stores can get into trouble in ways which an FS would never, do, ways
> so obvious we've never done tests for them. We know what the problems are:
> test for file and dir creation directly/indirectly under other files
> * mkdir(file/file)
> * mkdir(file/subdir)
> * dir under file/subdir/subdir
> * dir/dir2/file, verify dir & dir2 exist
> * dir/dir2/dir3, verify dir & dir2 exist
> * rename(src, file/dest)
> * rename(src, file/dir/dest)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15680) ITestNativeAzureFileSystemConcurrencyLive times out
[ https://issues.apache.org/jira/browse/HADOOP-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15680: Resolution: Fixed Fix Version/s: 3.1.2 Status: Resolved (was: Patch Available) committed to branch-3.1 & trunk: thanks! > ITestNativeAzureFileSystemConcurrencyLive times out > --- > > Key: HADOOP-15680 > URL: https://issues.apache.org/jira/browse/HADOOP-15680 > Project: Hadoop Common > Issue Type: Bug >Reporter: Andras Bokor >Assignee: Andras Bokor >Priority: Major > Fix For: 3.1.2 > > Attachments: HADOOP-15680.001.patch, HADOOP-15680.002.patch > > > When I am running tests locally ITestNativeAzureFileSystemConcurrencyLive > sometimes times out. > I would like to increase the timeout to avoid unnecessary noise. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15667) FileSystemMultipartUploader should verify that UploadHandle has non-0 length
[ https://issues.apache.org/jira/browse/HADOOP-15667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15667: Resolution: Fixed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) ran all tests against s3 ireland w s3guard/ddb, all well. committed to trunk > FileSystemMultipartUploader should verify that UploadHandle has non-0 length > > > Key: HADOOP-15667 > URL: https://issues.apache.org/jira/browse/HADOOP-15667 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Ewan Higgs >Assignee: Ewan Higgs >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15667.001.patch > > > The S3AMultipartUploader has a good check on the length of the UploadHandle. > This should be moved to MultipartUploader, made protected, and called in the > various implementations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15704) ABFS: Consider passing FS URI to CustomDelegationTokenManager
[ https://issues.apache.org/jira/browse/HADOOP-15704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597367#comment-16597367 ] Steve Loughran commented on HADOOP-15704: - bq. We currently have implementations of CustomDelegationTokenManager, and need to do a little leg work here, but it may be possible to update before GA. should be trivial in intellij; it's not like the property needs to be used —yet— > ABFS: Consider passing FS URI to CustomDelegationTokenManager > - > > Key: HADOOP-15704 > URL: https://issues.apache.org/jira/browse/HADOOP-15704 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Priority: Major > > Refer to Steve's comments in HADOOP-15692. Passing the FS or FS URI to the > CustomDelegationTokenManager would allow it to provide per-filesystem tokens. > We currently have implementations of CustomDelegationTokenManager, and need > to do a little leg work here, but it may be possible to update before GA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15687) Credentials class should allow access to aliases
[ https://issues.apache.org/jira/browse/HADOOP-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HADOOP-15687: -- Status: Patch Available (was: Open) Attached is a patch that adds two new methods to get to the full map of tokens and secret keys. It also changes the test (which was working around the lack of these methods) to make use of them. > Credentials class should allow access to aliases > > > Key: HADOOP-15687 > URL: https://issues.apache.org/jira/browse/HADOOP-15687 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Lars Francke >Assignee: Lars Francke >Priority: Trivial > Attachments: HADOOP-15687.patch > > > The credentials class can read token files from disk which are keyed by an > alias. It also allows to retrieve tokens by alias and it also allows to list > all tokens. > It does not - however - allow to get the full map of all tokens including the > aliases (or at least a list of all aliases). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15687) Credentials class should allow access to aliases
[ https://issues.apache.org/jira/browse/HADOOP-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke reassigned HADOOP-15687: - Assignee: Lars Francke > Credentials class should allow access to aliases > > > Key: HADOOP-15687 > URL: https://issues.apache.org/jira/browse/HADOOP-15687 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Lars Francke >Assignee: Lars Francke >Priority: Trivial > Attachments: HADOOP-15687.patch > > > The credentials class can read token files from disk which are keyed by an > alias. It also allows to retrieve tokens by alias and it also allows to list > all tokens. > It does not - however - allow to get the full map of all tokens including the > aliases (or at least a list of all aliases). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15687) Credentials class should allow access to aliases
[ https://issues.apache.org/jira/browse/HADOOP-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HADOOP-15687: -- Attachment: HADOOP-15687.patch > Credentials class should allow access to aliases > > > Key: HADOOP-15687 > URL: https://issues.apache.org/jira/browse/HADOOP-15687 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Lars Francke >Priority: Trivial > Attachments: HADOOP-15687.patch > > > The credentials class can read token files from disk which are keyed by an > alias. It also allows to retrieve tokens by alias and it also allows to list > all tokens. > It does not - however - allow to get the full map of all tokens including the > aliases (or at least a list of all aliases). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
