[jira] [Updated] (HADOOP-8299) ViewFileSystem link slash mount point crashes with IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/HADOOP-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HADOOP-8299: Fix Version/s: 2.10.1 > ViewFileSystem link slash mount point crashes with IndexOutOfBoundsException > > > Key: HADOOP-8299 > URL: https://issues.apache.org/jira/browse/HADOOP-8299 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Affects Versions: 3.0.0-alpha1 >Reporter: Eli Collins >Assignee: Manoj Govindassamy >Priority: Major > Fix For: 3.0.0-alpha2, 2.10.1 > > Attachments: HADOOP-8299.01.patch, HADOOP-8299.02.patch > > > We currently assume [a typical viewfs client > configuration|https://issues.apache.org/jira/secure/attachment/12507504/viewfs_TypicalMountTable.png] > is a set of non-overlapping mounts. This means every time you want to add a > new top-level directory you need to update the client-side mountable config. > If users could specify a slash mount, and then add additional mounts as > necessary they could add a new top-level directory without updating all > client configs (as long as the new top-level directory was being created on > the NN the slash mount points to). This could be achieved by HADOOP-8298 > (merge mounts, since we're effectively merging all new mount points with > slash) or having the notion of a "default NN" for a mount table. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18127) Backport HADOOP-13055 into branch-2.10
[ https://issues.apache.org/jira/browse/HADOOP-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17496262#comment-17496262 ] Konstantin Shvachko commented on HADOOP-18127: -- Back porting HADOOP-13055 needed two more commits. [PR #4015|https://github.com/apache/hadoop/pull/4015] includes three commits: * HADOOP-13722. Code cleanup -- ViewFileSystem and InodeTree. * HADOOP-12077. Provide a multi-URI replication Inode for ViewFs. * HADOOP-13055. Implement linkMergeSlash and linkFallback for ViewFileSystem > Backport HADOOP-13055 into branch-2.10 > -- > > Key: HADOOP-18127 > URL: https://issues.apache.org/jira/browse/HADOOP-18127 > Project: Hadoop Common > Issue Type: Sub-task > Components: viewfs >Affects Versions: 2.10.0 >Reporter: Konstantin Shvachko >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HADOOP-13055 introduce linkMergeSlash and linkFallback for ViewFileSystem. > Would be good to backport it to branch-2.10 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18109) Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems
[ https://issues.apache.org/jira/browse/HADOOP-18109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493099#comment-17493099 ] Konstantin Shvachko commented on HADOOP-18109: -- Committed to trunk and 3.* branches. Pending commit to 2.10 > Ensure that default permissions of directories under internal ViewFS > directories are the same as directories on target filesystems > -- > > Key: HADOOP-18109 > URL: https://issues.apache.org/jira/browse/HADOOP-18109 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Reporter: Chentao Yu >Assignee: Chentao Yu >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > * Ensure that default permissions of directories under internal ViewFS > directories are the same as directories on target filesystems > * Add new unit test -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18127) Backport HADOOP-13055 into branch-2.10
[ https://issues.apache.org/jira/browse/HADOOP-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HADOOP-18127: - Target Version/s: 2.10.2 Affects Version/s: 2.10.0 > Backport HADOOP-13055 into branch-2.10 > -- > > Key: HADOOP-18127 > URL: https://issues.apache.org/jira/browse/HADOOP-18127 > Project: Hadoop Common > Issue Type: Sub-task > Components: viewfs >Affects Versions: 2.10.0 >Reporter: Konstantin Shvachko >Priority: Major > > HADOOP-13055 introduce linkMergeSlash and linkFallback for ViewFileSystem. > Would be good to backport it to branch-2.10 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18127) Backport HADOOP-13055 into branch-2.10
Konstantin Shvachko created HADOOP-18127: Summary: Backport HADOOP-13055 into branch-2.10 Key: HADOOP-18127 URL: https://issues.apache.org/jira/browse/HADOOP-18127 Project: Hadoop Common Issue Type: Sub-task Components: viewfs Reporter: Konstantin Shvachko HADOOP-13055 introduce linkMergeSlash and linkFallback for ViewFileSystem. Would be good to backport it to branch-2.10 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18109) Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems
[ https://issues.apache.org/jira/browse/HADOOP-18109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17492882#comment-17492882 ] Konstantin Shvachko commented on HADOOP-18109: -- Linking jiras related to this. > Ensure that default permissions of directories under internal ViewFS > directories are the same as directories on target filesystems > -- > > Key: HADOOP-18109 > URL: https://issues.apache.org/jira/browse/HADOOP-18109 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Reporter: Chentao Yu >Assignee: Chentao Yu >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > * Ensure that default permissions of directories under internal ViewFS > directories are the same as directories on target filesystems > * Add new unit test -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18109) Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems
[ https://issues.apache.org/jira/browse/HADOOP-18109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HADOOP-18109: - Component/s: viewfs Target Version/s: 2.10.1 > Ensure that default permissions of directories under internal ViewFS > directories are the same as directories on target filesystems > -- > > Key: HADOOP-18109 > URL: https://issues.apache.org/jira/browse/HADOOP-18109 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Reporter: Chentao Yu >Assignee: Chentao Yu >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > * Ensure that default permissions of directories under internal ViewFS > directories are the same as directories on target filesystems > * Add new unit test -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17999) No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-17999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko resolved HADOOP-17999. -- Fix Version/s: 3.4.0 2.10.2 3.2.4 3.3.3 Hadoop Flags: Reviewed Resolution: Fixed I just committed this to all supported branches. Thank you [~abhishekd]. > No-op implementation of setWriteChecksum and setVerifyChecksum in > ViewFileSystem > > > Key: HADOOP-17999 > URL: https://issues.apache.org/jira/browse/HADOOP-17999 > Project: Hadoop Common > Issue Type: Bug >Reporter: Abhishek Das >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.4, 3.3.3 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently setVerifyChecksum and setWriteChecksum initializes all target file > systems which causes delay in hadoop shell copy commands such as get, put, > copyFromLocal etc. > This also eventually causes OOM. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17999) No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-17999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HADOOP-17999: - Summary: No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem (was: ViewFileSystem.setVerifyChecksum should not initialize all target filesystems) > No-op implementation of setWriteChecksum and setVerifyChecksum in > ViewFileSystem > > > Key: HADOOP-17999 > URL: https://issues.apache.org/jira/browse/HADOOP-17999 > Project: Hadoop Common > Issue Type: Bug >Reporter: Abhishek Das >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently setVerifyChecksum and setWriteChecksum initializes all target file > systems which causes delay in hadoop shell copy commands such as get, put, > copyFromLocal etc. > This also eventually causes OOM. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17999) ViewFileSystem.setVerifyChecksum should not initialize all target filesystems
[ https://issues.apache.org/jira/browse/HADOOP-17999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442389#comment-17442389 ] Konstantin Shvachko commented on HADOOP-17999: -- Was looking at the patch. Why do we need setting checksum in ViewFileSystem globally for all mount points? Seems it should be a per-target-file-system parameter, which is set when the mount point is initialized from its own config. > ViewFileSystem.setVerifyChecksum should not initialize all target filesystems > - > > Key: HADOOP-17999 > URL: https://issues.apache.org/jira/browse/HADOOP-17999 > Project: Hadoop Common > Issue Type: Bug >Reporter: Abhishek Das >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently setVerifyChecksum and setWriteChecksum initializes all target file > systems which causes delay in hadoop shell copy commands such as get, put, > copyFromLocal etc. > This also eventually causes OOM. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16532) Fix TestViewFsTrash to use the correct homeDir.
[
https://issues.apache.org/jira/browse/HADOOP-16532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-16532:
-
Component/s: (was: fs)
viewfs
> Fix TestViewFsTrash to use the correct homeDir.
> ---
>
> Key: HADOOP-16532
> URL: https://issues.apache.org/jira/browse/HADOOP-16532
> Project: Hadoop Common
> Issue Type: Bug
> Components: test, viewfs
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Xing Lin
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.3.2, 3.2.4
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> the test {{TestViewFsTrash}} uses the .Trash directory under the current
> user's home dir so
> * fails in test setups which block writing to it (jenkins)
> * fails when users have real trash in there
> * may fail if there are parallel test runs.
> the home dir should be under some test path of the build.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16532) Fix TestViewFsTrash to use the correct homeDir.
[
https://issues.apache.org/jira/browse/HADOOP-16532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko resolved HADOOP-16532.
--
Fix Version/s: 3.2.4
3.3.2
2.10.2
3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
I just committed this. Thank you [~xinglin].
> Fix TestViewFsTrash to use the correct homeDir.
> ---
>
> Key: HADOOP-16532
> URL: https://issues.apache.org/jira/browse/HADOOP-16532
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Xing Lin
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.3.2, 3.2.4
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> the test {{TestViewFsTrash}} uses the .Trash directory under the current
> user's home dir so
> * fails in test setups which block writing to it (jenkins)
> * fails when users have real trash in there
> * may fail if there are parallel test runs.
> the home dir should be under some test path of the build.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16532) Fix TestViewFsTrash to use the correct homeDir.
[
https://issues.apache.org/jira/browse/HADOOP-16532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-16532:
-
Summary: Fix TestViewFsTrash to use the correct homeDir. (was:
TestViewFsTrash uses home directory of real fs; brittle)
> Fix TestViewFsTrash to use the correct homeDir.
> ---
>
> Key: HADOOP-16532
> URL: https://issues.apache.org/jira/browse/HADOOP-16532
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Xing Lin
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> the test {{TestViewFsTrash}} uses the .Trash directory under the current
> user's home dir so
> * fails in test setups which block writing to it (jenkins)
> * fails when users have real trash in there
> * may fail if there are parallel test runs.
> the home dir should be under some test path of the build.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16532) TestViewFsTrash uses home directory of real fs; brittle
[
https://issues.apache.org/jira/browse/HADOOP-16532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425137#comment-17425137
]
Konstantin Shvachko commented on HADOOP-16532:
--
Well, Unit tests are hacks by definition. We mock variables and methods, inject
faults, set weird config values, etc., to achieve the desired "faulty" behavior.
You can try using {{setWorkingDirectory()}} is this feels less hacky to you.
> TestViewFsTrash uses home directory of real fs; brittle
> ---
>
> Key: HADOOP-16532
> URL: https://issues.apache.org/jira/browse/HADOOP-16532
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Xing Lin
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> the test {{TestViewFsTrash}} uses the .Trash directory under the current
> user's home dir so
> * fails in test setups which block writing to it (jenkins)
> * fails when users have real trash in there
> * may fail if there are parallel test runs.
> the home dir should be under some test path of the build.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16532) TestViewFsTrash uses home directory of real fs; brittle
[
https://issues.apache.org/jira/browse/HADOOP-16532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko reassigned HADOOP-16532:
Assignee: Xing Lin
> TestViewFsTrash uses home directory of real fs; brittle
> ---
>
> Key: HADOOP-16532
> URL: https://issues.apache.org/jira/browse/HADOOP-16532
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Xing Lin
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> the test {{TestViewFsTrash}} uses the .Trash directory under the current
> user's home dir so
> * fails in test setups which block writing to it (jenkins)
> * fails when users have real trash in there
> * may fail if there are parallel test runs.
> the home dir should be under some test path of the build.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16532) TestViewFsTrash uses home directory of real fs; brittle
[
https://issues.apache.org/jira/browse/HADOOP-16532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424310#comment-17424310
]
Konstantin Shvachko commented on HADOOP-16532:
--
This definitely fixes the test, but the cost seems high.
Can't we just add something like
{code}
@BeforeClass
static public void init() {
System.setProperty("user.home", "./target");
}
{code}
And probably delete {{target/.Trash}} in {{@AfterClass}}.
> TestViewFsTrash uses home directory of real fs; brittle
> ---
>
> Key: HADOOP-16532
> URL: https://issues.apache.org/jira/browse/HADOOP-16532
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> the test {{TestViewFsTrash}} uses the .Trash directory under the current
> user's home dir so
> * fails in test setups which block writing to it (jenkins)
> * fails when users have real trash in there
> * may fail if there are parallel test runs.
> the home dir should be under some test path of the build.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17853) ABFS: Enable optional store connectivity over azure specific protocol for data egress
[ https://issues.apache.org/jira/browse/HADOOP-17853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17404092#comment-17404092 ] Konstantin Shvachko commented on HADOOP-17853: -- ??conflicts to trunk?? I think I understand what is going on. I am trying to apply the [patch-diff generated by github|https://patch-diff.githubusercontent.com/raw/apache/hadoop/pull/3309.patch] to current trunk. But since you were occasionally merging trunk into your branch, rather than rebasing it, different patches were applied to different historical states of trunk, which conflict with the current state. A better way is to rebase your branch on trunk before each commit and avoid merge commits. [~snvijaya], also wanted to ask you to post some benchmark data for expected improvements in IO performance using native Azure protocols. As a motivation for such a large new feature. > ABFS: Enable optional store connectivity over azure specific protocol for > data egress > - > > Key: HADOOP-17853 > URL: https://issues.apache.org/jira/browse/HADOOP-17853 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > This Jira is to provide an option to enable store access on read path over an > Azure specific protocol. This will only work on Azure VMs and hence will be > disabled by default. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17853) ABFS: Enable optional store connectivity over azure specific protocol for data egress
[ https://issues.apache.org/jira/browse/HADOOP-17853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17402446#comment-17402446 ] Konstantin Shvachko commented on HADOOP-17853: -- Hi [~snvijaya]. Thank you for the patch. Just wanted to mention this is a 1.2 MB patch. It will be very hard to review it at once. I see there is a lot of refactoring going on there. It would be good to split it into several PRs to ease the review process. Also I tried to apply the PR to current trunk and got some conflicts. You might need to rebase your branch on current trunk. > ABFS: Enable optional store connectivity over azure specific protocol for > data egress > - > > Key: HADOOP-17853 > URL: https://issues.apache.org/jira/browse/HADOOP-17853 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to provide an option to enable store access on read path over an > Azure specific protocol. This will only work on Azure VMs and hence will be > disabled by default. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17853) ABFS: Enable optional store connectivity over azure specific protocol for data egress
[ https://issues.apache.org/jira/browse/HADOOP-17853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reassigned HADOOP-17853: Assignee: Sneha Vijayarajan (was: Konstantin Shvachko) > ABFS: Enable optional store connectivity over azure specific protocol for > data egress > - > > Key: HADOOP-17853 > URL: https://issues.apache.org/jira/browse/HADOOP-17853 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to provide an option to enable store access on read path over an > Azure specific protocol. This will only work on Azure VMs and hence will be > disabled by default. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17853) ABFS: Enable optional store connectivity over azure specific protocol for data egress
[ https://issues.apache.org/jira/browse/HADOOP-17853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reassigned HADOOP-17853: Assignee: Konstantin Shvachko (was: Sneha Vijayarajan) > ABFS: Enable optional store connectivity over azure specific protocol for > data egress > - > > Key: HADOOP-17853 > URL: https://issues.apache.org/jira/browse/HADOOP-17853 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0 >Reporter: Sneha Vijayarajan >Assignee: Konstantin Shvachko >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to provide an option to enable store access on read path over an > Azure specific protocol. This will only work on Azure VMs and hence will be > disabled by default. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17819) Add extensions to ProtobufRpcEngine RequestHeaderProto
[ https://issues.apache.org/jira/browse/HADOOP-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko resolved HADOOP-17819. -- Fix Version/s: 3.2.3 2.10.2 3.4.0 Hadoop Flags: Reviewed Assignee: Hector Sandoval Chaverri Resolution: Fixed I just committed this. Thank you [~hchaverri] > Add extensions to ProtobufRpcEngine RequestHeaderProto > -- > > Key: HADOOP-17819 > URL: https://issues.apache.org/jira/browse/HADOOP-17819 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.3 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The header used in ProtobufRpcEngine messages doesn't allow for new > properties to be added by child classes. We can add a range of extensions > that can be useful for proto classes that need to extend RequestHeaderProto. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[ https://issues.apache.org/jira/browse/HADOOP-17028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko resolved HADOOP-17028. -- Fix Version/s: 3.3.2 3.2.3 2.10.2 3.4.0 3.1.5 Hadoop Flags: Reviewed Resolution: Fixed > ViewFS should initialize target filesystems lazily > -- > > Key: HADOOP-17028 > URL: https://issues.apache.org/jira/browse/HADOOP-17028 > Project: Hadoop Common > Issue Type: Bug > Components: client-mounts, fs, viewfs >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Fix For: 3.1.5, 3.4.0, 2.10.2, 3.2.3, 3.3.2 > > Time Spent: 7h 50m > Remaining Estimate: 0h > > Currently viewFS initialize all configured target filesystems when > viewfs#init itself. > Some target file system initialization involve creating heavy objects and > proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient > object which will create proxy connections to NN etc. > For example: if ViewFS configured with 10 target fs with hdfs uri and 2 > targets with s3a. > If one of the client only work with s3a target, But ViewFS will initialize > all targets irrespective of what clients interested to work with. That means, > here client will create 10 DFS initializations and 2 s3a initializations. Its > unnecessary to have DFS initialization here. So, it will be a good idea to > initialize the target fs only when first time usage call come to particular > target fs scheme. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[ https://issues.apache.org/jira/browse/HADOOP-17028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384553#comment-17384553 ] Konstantin Shvachko commented on HADOOP-17028: -- Committed PR #3218 to branch-2.10. Thank you [~abhishekd] > ViewFS should initialize target filesystems lazily > -- > > Key: HADOOP-17028 > URL: https://issues.apache.org/jira/browse/HADOOP-17028 > Project: Hadoop Common > Issue Type: Bug > Components: client-mounts, fs, viewfs >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Time Spent: 7h 50m > Remaining Estimate: 0h > > Currently viewFS initialize all configured target filesystems when > viewfs#init itself. > Some target file system initialization involve creating heavy objects and > proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient > object which will create proxy connections to NN etc. > For example: if ViewFS configured with 10 target fs with hdfs uri and 2 > targets with s3a. > If one of the client only work with s3a target, But ViewFS will initialize > all targets irrespective of what clients interested to work with. That means, > here client will create 10 DFS initializations and 2 s3a initializations. Its > unnecessary to have DFS initialization here. So, it will be a good idea to > initialize the target fs only when first time usage call come to particular > target fs scheme. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[ https://issues.apache.org/jira/browse/HADOOP-17028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380242#comment-17380242 ] Konstantin Shvachko commented on HADOOP-17028: -- I committed this to trunk and branches 3.1, 3.2, 3.3. Thanks [~abhishekd] for working on it. Will need a separate patch for branch-2.10. Too many conflicts. > ViewFS should initialize target filesystems lazily > -- > > Key: HADOOP-17028 > URL: https://issues.apache.org/jira/browse/HADOOP-17028 > Project: Hadoop Common > Issue Type: Bug > Components: client-mounts, fs, viewfs >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Time Spent: 7h > Remaining Estimate: 0h > > Currently viewFS initialize all configured target filesystems when > viewfs#init itself. > Some target file system initialization involve creating heavy objects and > proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient > object which will create proxy connections to NN etc. > For example: if ViewFS configured with 10 target fs with hdfs uri and 2 > targets with s3a. > If one of the client only work with s3a target, But ViewFS will initialize > all targets irrespective of what clients interested to work with. That means, > here client will create 10 DFS initializations and 2 s3a initializations. Its > unnecessary to have DFS initialization here. So, it will be a good idea to > initialize the target fs only when first time usage call come to particular > target fs scheme. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[
https://issues.apache.org/jira/browse/HADOOP-17028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370329#comment-17370329
]
Konstantin Shvachko commented on HADOOP-17028:
--
Steve, this looks like incompatible change as it replaced the parameter to a
different class:
{code}
public static CompletableFuture eval(
- FunctionsRaisingIOE.CallableRaisingIOE callable) {
+ CallableRaisingIOE callable) {
CompletableFuture result = new CompletableFuture<>();
{code}
Also introduction of dead code should have been avoided. Looking at the
history, you introduced
{{org.apache.hadoop.fs.impl.FunctionsRaisingIOE.FunctionRaisingIOE}} as a part
of HADOOP-15183. But it wasn't used anywhere. Then HADOOP-17450 deprecated it.
One good thing about it that you can remove the deprecated classes as they have
never been used.
Supporting dead code is just a waste of energy it also makes back porting
really hard. You don't seem to care about older versions, but many people do.
> ViewFS should initialize target filesystems lazily
> --
>
> Key: HADOOP-17028
> URL: https://issues.apache.org/jira/browse/HADOOP-17028
> Project: Hadoop Common
> Issue Type: Bug
> Components: client-mounts, fs, viewfs
>Affects Versions: 3.2.1
>Reporter: Uma Maheswara Rao G
>Assignee: Abhishek Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> Currently viewFS initialize all configured target filesystems when
> viewfs#init itself.
> Some target file system initialization involve creating heavy objects and
> proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient
> object which will create proxy connections to NN etc.
> For example: if ViewFS configured with 10 target fs with hdfs uri and 2
> targets with s3a.
> If one of the client only work with s3a target, But ViewFS will initialize
> all targets irrespective of what clients interested to work with. That means,
> here client will create 10 DFS initializations and 2 s3a initializations. Its
> unnecessary to have DFS initialization here. So, it will be a good idea to
> initialize the target fs only when first time usage call come to particular
> target fs scheme.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[
https://issues.apache.org/jira/browse/HADOOP-17028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357581#comment-17357581
]
Konstantin Shvachko commented on HADOOP-17028:
--
I'll re-post here mu comment from the linked PR for visibility.
It looks like [~ste...@apache.org] deprecated this class later after his
comment. Don't know what prompted the refactoring, but
{{org.apache.hadoop.fs.impl.FunctionsRaisingIOE.FunctionRaisingIOE}}
should be now
{{org.apache.hadoop.util.functional.FunctionRaisingIOE}}
Moving interfaces, which are public by default, from one package to another is
considered an incompatible change, especially since the previous variant had
been released. Besides, it has been committed only to branch-3.3 contributing
to further divergence of the supported branches and making backporting yet
harder.
So [~abhishekd], I prefer to avoid using {{FunctionRaisingIOE}} in this patch
if possible. This will simplify the backport and avoid using unstable APIs.
> ViewFS should initialize target filesystems lazily
> --
>
> Key: HADOOP-17028
> URL: https://issues.apache.org/jira/browse/HADOOP-17028
> Project: Hadoop Common
> Issue Type: Bug
> Components: client-mounts, fs, viewfs
>Affects Versions: 3.2.1
>Reporter: Uma Maheswara Rao G
>Assignee: Abhishek Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Currently viewFS initialize all configured target filesystems when
> viewfs#init itself.
> Some target file system initialization involve creating heavy objects and
> proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient
> object which will create proxy connections to NN etc.
> For example: if ViewFS configured with 10 target fs with hdfs uri and 2
> targets with s3a.
> If one of the client only work with s3a target, But ViewFS will initialize
> all targets irrespective of what clients interested to work with. That means,
> here client will create 10 DFS initializations and 2 s3a initializations. Its
> unnecessary to have DFS initialization here. So, it will be a good idea to
> initialize the target fs only when first time usage call come to particular
> target fs scheme.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[
https://issues.apache.org/jira/browse/HADOOP-17028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357581#comment-17357581
]
Konstantin Shvachko edited comment on HADOOP-17028 at 6/4/21, 7:19 PM:
---
I'll re-post here my comment from the linked PR for visibility.
It looks like [~ste...@apache.org] deprecated this class later after his
comment. Don't know what prompted the refactoring, but
{{org.apache.hadoop.fs.impl.FunctionsRaisingIOE.FunctionRaisingIOE}}
should be now
{{org.apache.hadoop.util.functional.FunctionRaisingIOE}}
Moving interfaces, which are public by default, from one package to another is
considered an incompatible change, especially since the previous variant had
been released. Besides, it has been committed only to branch-3.3 contributing
to further divergence of the supported branches and making backporting yet
harder.
So [~abhishekd], I prefer to avoid using {{FunctionRaisingIOE}} in this patch
if possible. This will simplify the backport and avoid using unstable APIs.
was (Author: shv):
I'll re-post here mu comment from the linked PR for visibility.
It looks like [~ste...@apache.org] deprecated this class later after his
comment. Don't know what prompted the refactoring, but
{{org.apache.hadoop.fs.impl.FunctionsRaisingIOE.FunctionRaisingIOE}}
should be now
{{org.apache.hadoop.util.functional.FunctionRaisingIOE}}
Moving interfaces, which are public by default, from one package to another is
considered an incompatible change, especially since the previous variant had
been released. Besides, it has been committed only to branch-3.3 contributing
to further divergence of the supported branches and making backporting yet
harder.
So [~abhishekd], I prefer to avoid using {{FunctionRaisingIOE}} in this patch
if possible. This will simplify the backport and avoid using unstable APIs.
> ViewFS should initialize target filesystems lazily
> --
>
> Key: HADOOP-17028
> URL: https://issues.apache.org/jira/browse/HADOOP-17028
> Project: Hadoop Common
> Issue Type: Bug
> Components: client-mounts, fs, viewfs
>Affects Versions: 3.2.1
>Reporter: Uma Maheswara Rao G
>Assignee: Abhishek Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Currently viewFS initialize all configured target filesystems when
> viewfs#init itself.
> Some target file system initialization involve creating heavy objects and
> proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient
> object which will create proxy connections to NN etc.
> For example: if ViewFS configured with 10 target fs with hdfs uri and 2
> targets with s3a.
> If one of the client only work with s3a target, But ViewFS will initialize
> all targets irrespective of what clients interested to work with. That means,
> here client will create 10 DFS initializations and 2 s3a initializations. Its
> unnecessary to have DFS initialization here. So, it will be a good idea to
> initialize the target fs only when first time usage call come to particular
> target fs scheme.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[ https://issues.apache.org/jira/browse/HADOOP-17028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17356063#comment-17356063 ] Konstantin Shvachko commented on HADOOP-17028: -- Left minor comments on the PR. The approach looks reasonable to me. The important thing is to get a Jenkins build. Looks like it could not run. May be you fork got stale. You should probably rebase on current trunk. > ViewFS should initialize target filesystems lazily > -- > > Key: HADOOP-17028 > URL: https://issues.apache.org/jira/browse/HADOOP-17028 > Project: Hadoop Common > Issue Type: Bug > Components: client-mounts, fs, viewfs >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Time Spent: 3h 40m > Remaining Estimate: 0h > > Currently viewFS initialize all configured target filesystems when > viewfs#init itself. > Some target file system initialization involve creating heavy objects and > proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient > object which will create proxy connections to NN etc. > For example: if ViewFS configured with 10 target fs with hdfs uri and 2 > targets with s3a. > If one of the client only work with s3a target, But ViewFS will initialize > all targets irrespective of what clients interested to work with. That means, > here client will create 10 DFS initializations and 2 s3a initializations. Its > unnecessary to have DFS initialization here. So, it will be a good idea to > initialize the target fs only when first time usage call come to particular > target fs scheme. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17680) Allow ProtobufRpcEngine to be extensible
[ https://issues.apache.org/jira/browse/HADOOP-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340499#comment-17340499 ] Konstantin Shvachko commented on HADOOP-17680: -- I committed this to trunk and branches 3.* [~hchaverri] could you please create a patch for branch-2.10 as well. > Allow ProtobufRpcEngine to be extensible > > > Key: HADOOP-17680 > URL: https://issues.apache.org/jira/browse/HADOOP-17680 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > The ProtobufRpcEngine class doesn't allow for new RpcEngine implementations > to extend some of its inner classes (e.g. Invoker and > Server.ProtoBufRpcInvoker). Also, some of its methods are long enough such > that overriding them would result in a lot of code duplication (e.g. > Invoker#invoke and Server.ProtoBufRpcInvoker#call). > When implementing a new RpcEngine, it would be helpful to reuse most of the > code already in ProtobufRpcEngine. This would allow new fields to be added to > the RPC header or message with minimal code changes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17680) Allow ProtobufRpcEngine to be extensible
[ https://issues.apache.org/jira/browse/HADOOP-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338697#comment-17338697 ] Konstantin Shvachko commented on HADOOP-17680: -- There isn't much you can do about existing constructors. +1 for the latest patch. > Allow ProtobufRpcEngine to be extensible > > > Key: HADOOP-17680 > URL: https://issues.apache.org/jira/browse/HADOOP-17680 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > The ProtobufRpcEngine class doesn't allow for new RpcEngine implementations > to extend some of its inner classes (e.g. Invoker and > Server.ProtoBufRpcInvoker). Also, some of its methods are long enough such > that overriding them would result in a lot of code duplication (e.g. > Invoker#invoke and Server.ProtoBufRpcInvoker#call). > When implementing a new RpcEngine, it would be helpful to reuse most of the > code already in ProtobufRpcEngine. This would allow new fields to be added to > the RPC header or message with minimal code changes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10335) An ip whitelist based implementation to resolve Sasl properties per connection
[ https://issues.apache.org/jira/browse/HADOOP-10335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HADOOP-10335: - Summary: An ip whitelist based implementation to resolve Sasl properties per connection (was: An ip whilelist based implementation to resolve Sasl properties per connection) > An ip whitelist based implementation to resolve Sasl properties per connection > -- > > Key: HADOOP-10335 > URL: https://issues.apache.org/jira/browse/HADOOP-10335 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Benoy Antony >Assignee: Benoy Antony >Priority: Major > Fix For: 2.6.0 > > Attachments: HADOOP-10335.patch, HADOOP-10335.patch, > HADOOP-10335.patch, HADOOP-10335.patch, HADOOP-10335.patch, HADOOP-10335.pdf > > > As noted in HADOOP-10221, it is sometimes required for a Hadoop Server to > communicate with some client over encrypted channel and with some other > clients over unencrypted channel. > Hadoop-10221 introduced an interface _SaslPropertiesResolver_ and the > changes required to plugin and use _SaslPropertiesResolver_ to identify the > SaslProperties to be used for a connection. > In this jira, an ip-whitelist based implementation of > _SaslPropertiesResolver_ is attempted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17501) Fix logging typo in ShutdownHookManager
[
https://issues.apache.org/jira/browse/HADOOP-17501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-17501:
-
Description: Three log messages in {{ShutdownHookManager}} have a typo,
saying "ShutdownHookManger". Should be "ShutdownHookManager" (was: Three log
messages in {{ShutdownHookManager}} have a typo, saying "ShutdownHookManger".
SHould be "ShutdownHookManager")
> Fix logging typo in ShutdownHookManager
> ---
>
> Key: HADOOP-17501
> URL: https://issues.apache.org/jira/browse/HADOOP-17501
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: Konstantin Shvachko
>Priority: Major
> Labels: newbie
>
> Three log messages in {{ShutdownHookManager}} have a typo, saying
> "ShutdownHookManger". Should be "ShutdownHookManager"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17501) Fix logging typo in ShutdownHookManager
Konstantin Shvachko created HADOOP-17501:
Summary: Fix logging typo in ShutdownHookManager
Key: HADOOP-17501
URL: https://issues.apache.org/jira/browse/HADOOP-17501
Project: Hadoop Common
Issue Type: Improvement
Components: common
Reporter: Konstantin Shvachko
Three log messages in {{ShutdownHookManager}} have a typo, saying
"ShutdownHookManger". SHould be "ShutdownHookManager"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17477) [SBN read] Implement msync() for ViewFS
[
https://issues.apache.org/jira/browse/HADOOP-17477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-17477:
-
Description:
We should implement {{FileSystem.msync()}} for {{ViewFS}} and
{{ViewFileSystem}}.
It should call msync() on all volumes for which observer reads are enabled and
{{dfs.namenode.state.context.enabled = true}}
was:
We should implement {{FileSystem.mscyn()}} for {{ViewFS}} and
{{ViewFileSystem}}.
It should call msync() on all volumes for which observer reads are enabled and
{{dfs.namenode.state.context.enabled = true}}
> [SBN read] Implement msync() for ViewFS
> ---
>
> Key: HADOOP-17477
> URL: https://issues.apache.org/jira/browse/HADOOP-17477
> Project: Hadoop Common
> Issue Type: Improvement
> Components: viewfs
>Reporter: Konstantin Shvachko
>Assignee: Abhishek Das
>Priority: Major
>
> We should implement {{FileSystem.msync()}} for {{ViewFS}} and
> {{ViewFileSystem}}.
> It should call msync() on all volumes for which observer reads are enabled
> and {{dfs.namenode.state.context.enabled = true}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17477) [SBN read] Implement msync() for ViewFS
[
https://issues.apache.org/jira/browse/HADOOP-17477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko reassigned HADOOP-17477:
Assignee: Abhishek Das
> [SBN read] Implement msync() for ViewFS
> ---
>
> Key: HADOOP-17477
> URL: https://issues.apache.org/jira/browse/HADOOP-17477
> Project: Hadoop Common
> Issue Type: Improvement
> Components: viewfs
>Reporter: Konstantin Shvachko
>Assignee: Abhishek Das
>Priority: Major
>
> We should implement {{FileSystem.mscyn()}} for {{ViewFS}} and
> {{ViewFileSystem}}.
> It should call msync() on all volumes for which observer reads are enabled
> and {{dfs.namenode.state.context.enabled = true}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17477) [SBN read] Implement msync() for ViewFS
Konstantin Shvachko created HADOOP-17477:
Summary: [SBN read] Implement msync() for ViewFS
Key: HADOOP-17477
URL: https://issues.apache.org/jira/browse/HADOOP-17477
Project: Hadoop Common
Issue Type: Improvement
Components: viewfs
Reporter: Konstantin Shvachko
We should implement {{FileSystem.mscyn()}} for {{ViewFS}} and
{{ViewFileSystem}}.
It should call msync() on all volumes for which observer reads are enabled and
{{dfs.namenode.state.context.enabled = true}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16208) Do Not Log InterruptedException in Client
[
https://issues.apache.org/jira/browse/HADOOP-16208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-16208:
-
Fix Version/s: 2.10.2
> Do Not Log InterruptedException in Client
> -
>
> Key: HADOOP-16208
> URL: https://issues.apache.org/jira/browse/HADOOP-16208
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 3.3.0, 3.2.1, 3.1.3, 2.10.2
>
> Attachments: HADOOP-16208.1.patch, HADOOP-16208.2.patch,
> HADOOP-16208.3.patch
>
>
> {code:java}
> } catch (InterruptedException e) {
> Thread.currentThread().interrupt();
> LOG.warn("interrupted waiting to send rpc request to server", e);
> throw new IOException(e);
> }
> {code}
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L1450
> I'm working on a project that uses an {{ExecutorService}} to launch a bunch
> of threads. Each thread spins up an HDFS client connection. At any point in
> time, the program can terminate and call {{ExecutorService#shutdownNow()}} to
> forcibly close vis-à-vis {{Thread#interrupt()}}. At that point, I get a
> cascade of logging from the above code and there's no easy to way to turn it
> off.
> "Log and throw" is generally frowned upon, just throw the {{Exception}} and
> move on.
> https://community.oracle.com/docs/DOC-983543#logAndThrow
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16208) Do Not Log InterruptedException in Client
[
https://issues.apache.org/jira/browse/HADOOP-16208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253662#comment-17253662
]
Konstantin Shvachko commented on HADOOP-16208:
--
I see this exception regularly in StandbyNode and ObserverNode logs. When SBN
does journal tailing as a part of fast tailing SBN makes a quorum call to QJM.
When it receives enough responses it legitimately cancels the remaining
unfinished calls. This is when we get this exception printed. See
{{o.a.h.ipc.Client.call()}} and
{{QuorumJournalManager.selectRpcInputStreams()}} where it invokes
{{q.cancelCalls()}}. Looks like this:
{noformat:nowrap}
2020-12-22 18:00:26,690 WARN org.apache.hadoop.ipc.Client: interrupted waiting
to send rpc request to server
java.lang.InterruptedException
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at
org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1162)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy12.getJournaledEdits(Unknown Source)
at
org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolTranslatorPB.getJournaledEdits(QJournalProtocolTranslatorPB.java:268)
at
org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$13.call(IPCLoggerChannel.java:562)
at
org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$13.call(IPCLoggerChannel.java:559)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
Not a bug by itself but makes people worry if there is a problem, and it is
pretty annoying. Will cherry pick into 2.10 as well.
> Do Not Log InterruptedException in Client
> -
>
> Key: HADOOP-16208
> URL: https://issues.apache.org/jira/browse/HADOOP-16208
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HADOOP-16208.1.patch, HADOOP-16208.2.patch,
> HADOOP-16208.3.patch
>
>
> {code:java}
> } catch (InterruptedException e) {
> Thread.currentThread().interrupt();
> LOG.warn("interrupted waiting to send rpc request to server", e);
> throw new IOException(e);
> }
> {code}
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L1450
> I'm working on a project that uses an {{ExecutorService}} to launch a bunch
> of threads. Each thread spins up an HDFS client connection. At any point in
> time, the program can terminate and call {{ExecutorService#shutdownNow()}} to
> forcibly close vis-à-vis {{Thread#interrupt()}}. At that point, I get a
> cascade of logging from the above code and there's no easy to way to turn it
> off.
> "Log and throw" is generally frowned upon, just throw the {{Exception}} and
> move on.
> https://community.oracle.com/docs/DOC-983543#logAndThrow
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17336) Backport HADOOP-16005-"NativeAzureFileSystem does not support setXAttr" and HADOOP-16785. "Improve wasb and abfs resilience on double close() calls. followup to abfs
[ https://issues.apache.org/jira/browse/HADOOP-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225612#comment-17225612 ] Konstantin Shvachko commented on HADOOP-17336: -- +1 the backport looks good to me. > Backport HADOOP-16005-"NativeAzureFileSystem does not support setXAttr" and > HADOOP-16785. "Improve wasb and abfs resilience on double close() calls. > followup to abfs close() fix." to branch-2.10 > -- > > Key: HADOOP-17336 > URL: https://issues.apache.org/jira/browse/HADOOP-17336 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.10.1 >Reporter: Sally Zuo >Assignee: Sally Zuo >Priority: Major > Attachments: Hadoop-17336-branch-2.10.001.patch, > Hadoop-17336-branch-2.10.002.patch > > > Backport: > HADOOP-16005-"NativeAzureFileSystem does not support setXAttr" to branch-2.10 > HADOOP-16785. "Improve wasb and abfs resilience on double close() calls. > followup to abfs close() fix." > from branch-3.2 to branch-2.10 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17010) Add queue capacity weights support in FairCallQueue
[
https://issues.apache.org/jira/browse/HADOOP-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098089#comment-17098089
]
Konstantin Shvachko commented on HADOOP-17010:
--
Hey [~fengnanli] this broke {{TestRefreshCallQueue}} in HDFS, see HDFS-15325.
You might wan to check any other CallQueue classes that are out there.
> Add queue capacity weights support in FairCallQueue
> ---
>
> Key: HADOOP-17010
> URL: https://issues.apache.org/jira/browse/HADOOP-17010
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Major
> Attachments: HADOOP-17010.001.patch, HADOOP-17010.002.patch
>
>
> Right now in FairCallQueue all subqueues share the same capacity by evenly
> distributing total capacity. This requested feature is to make subqueues able
> to have different queue capacity where more important queues can have more
> capacity, thus less queue overflow and client backoffs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17026) TestRefreshCallQueue is failing due to changed CallQueue constructor
Konstantin Shvachko created HADOOP-17026: Summary: TestRefreshCallQueue is failing due to changed CallQueue constructor Key: HADOOP-17026 URL: https://issues.apache.org/jira/browse/HADOOP-17026 Project: Hadoop Common Issue Type: Bug Components: test Reporter: Konstantin Shvachko -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15711) Fix branch-2 builds
[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758813#comment-16758813
]
Konstantin Shvachko commented on HADOOP-15711:
--
I don't think we should drastically change the dependency for branch-2 from
Java 7 to Java 8.
We only want to fix the precommit test builds on Jenkins to use openJDK-8. When
we release branch-2 we should still use Java 7 to generate binaries, and
compatibility with Java 7 should not be affected.
EOL for branch-2 will just mean that everybody will switch to their local
branches, as all major clusters still run on Hadoop 2. This will be
self-destructive for everybody: the community, vendors, all consumers of
Hadoop. I suggest we just focus on fixing the builds here as the jira states.
> Fix branch-2 builds
> ---
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Jonathan Hung
>Priority: Critical
> Attachments: HADOOP-15711.001.branch-2.patch
>
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-11890) Uber-JIRA: Hadoop should support IPv6
[
https://issues.apache.org/jira/browse/HADOOP-11890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757868#comment-16757868
]
Konstantin Shvachko edited comment on HADOOP-11890 at 2/1/19 1:53 AM:
--
Hey [~templedf]. Her is the latest I heard from Gary Helmling.
{quote}pushed up a cleaned up version of our branch with only the IPv6 patches
on top of 2.7.3 to github:
[https://github.com/ghelmling/hadoop/tree/branch-2.7.3-ipv6]
A full list of the commits applied in the branch is attached.
This is basically the same set of changes that we are currently running.
There are still some test issues to work out though. I've seen a number of
tests fail unless -Djava.net.preferIPv4Stack=true is used.
{quote}
Attaching his change log too: [^hadoop_2.7.3_ipv6_commits.txt].
This was done for branch 2.7 back then. I don't know how much effort is needed
to port it to trunk.
Also did not check of the github
[branch-2.7.3-ipv6|https://github.com/ghelmling/hadoop/tree/branch-2.7.3-ipv6]
and the apache [HADOOP-11890
branch|https://github.com/apache/hadoop/tree/HADOOP-11890] are the same, may be
not.
Hope this helps
was (Author: shv):
Hey [~templedf]. Her is the latest I heard from Gary Helmling.
{quote}pushed up a cleaned up version of our branch with only the IPv6 patches
on top of 2.7.3 to github:
[https://github.com/ghelmling/hadoop/tree/branch-2.7.3-ipv6]
A full list of the commits applied in the branch is attached.
This is basically the same set of changes that we are currently running.
There are still some test issues to work out though. I've seen a number of
tests fail unless -Djava.net.preferIPv4Stack=true is used.
{quote}
Attaching his change log too: [^hadoop_2.7.3_ipv6_commits.txt].
This was done for branch 2.7 back then. I don't know how much effort is needed
to port it to trunk.
Hope this helps
> Uber-JIRA: Hadoop should support IPv6
> -
>
> Key: HADOOP-11890
> URL: https://issues.apache.org/jira/browse/HADOOP-11890
> Project: Hadoop Common
> Issue Type: Improvement
> Components: net
>Reporter: Nate Edel
>Assignee: Nate Edel
>Priority: Major
> Labels: ipv6
> Attachments: hadoop_2.7.3_ipv6_commits.txt
>
>
> Hadoop currently treats IPv6 as unsupported. Track related smaller issues to
> support IPv6.
> (Current case here is mainly HBase on HDFS, so any suggestions about other
> test cases/workload are really appreciated.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11890) Uber-JIRA: Hadoop should support IPv6
[
https://issues.apache.org/jira/browse/HADOOP-11890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757868#comment-16757868
]
Konstantin Shvachko commented on HADOOP-11890:
--
Hey [~templedf]. Her is the latest I heard from Gary Helmling.
{quote}pushed up a cleaned up version of our branch with only the IPv6 patches
on top of 2.7.3 to github:
[https://github.com/ghelmling/hadoop/tree/branch-2.7.3-ipv6]
A full list of the commits applied in the branch is attached.
This is basically the same set of changes that we are currently running.
There are still some test issues to work out though. I've seen a number of
tests fail unless -Djava.net.preferIPv4Stack=true is used.
{quote}
Attaching his change log too: [^hadoop_2.7.3_ipv6_commits.txt].
This was done for branch 2.7 back then. I don't know how much effort is needed
to port it to trunk.
Hope this helps
> Uber-JIRA: Hadoop should support IPv6
> -
>
> Key: HADOOP-11890
> URL: https://issues.apache.org/jira/browse/HADOOP-11890
> Project: Hadoop Common
> Issue Type: Improvement
> Components: net
>Reporter: Nate Edel
>Assignee: Nate Edel
>Priority: Major
> Labels: ipv6
> Attachments: hadoop_2.7.3_ipv6_commits.txt
>
>
> Hadoop currently treats IPv6 as unsupported. Track related smaller issues to
> support IPv6.
> (Current case here is mainly HBase on HDFS, so any suggestions about other
> test cases/workload are really appreciated.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11890) Uber-JIRA: Hadoop should support IPv6
[ https://issues.apache.org/jira/browse/HADOOP-11890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HADOOP-11890: - Attachment: hadoop_2.7.3_ipv6_commits.txt > Uber-JIRA: Hadoop should support IPv6 > - > > Key: HADOOP-11890 > URL: https://issues.apache.org/jira/browse/HADOOP-11890 > Project: Hadoop Common > Issue Type: Improvement > Components: net >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Major > Labels: ipv6 > Attachments: hadoop_2.7.3_ipv6_commits.txt > > > Hadoop currently treats IPv6 as unsupported. Track related smaller issues to > support IPv6. > (Current case here is mainly HBase on HDFS, so any suggestions about other > test cases/workload are really appreciated.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746589#comment-16746589 ] Konstantin Shvachko commented on HADOOP-15481: -- Hey [~vagarychen], if you commit to branch-2, then you should also commit to all branches above it, that is 3.0, 3.1, 3.2. And then update fix version(s). > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15481-branch-2.003.patch, HADOOP-15481.001.patch, > HADOOP-15481.001.patch, HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15711) Fix branch-2 builds
[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623935#comment-16623935
]
Konstantin Shvachko commented on HADOOP-15711:
--
[~aw] thanks a lot for your insight, this helps understanding the problem and
differences between the branches.
Looked at the PreCommit runs with the attached patch more closely. Definitely
some cheating there, as many tests were not executed at all (Just as [~aw]
commented). This explains fast running time.
I also made Jenkins [execute branch-3.1
build|https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/12/console],
which completed in 12 hours.
I don't see any shortcut solutions to the branch-2 build problem at this point.
We should probably take the long path and understand the fixes that were done
to unit tests on branch-3, that were not backported to branch-2.
> Fix branch-2 builds
> ---
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Jonathan Hung
>Priority: Critical
> Attachments: HADOOP-15711.001.branch-2.patch
>
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15711) Fix branch-2 builds
[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621256#comment-16621256
]
Konstantin Shvachko edited comment on HADOOP-15711 at 9/19/18 10:07 PM:
[Ran a build for
branch-2.7|https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/11/],
which doesn't have the HADOOP-13514 backport. It was aborted after 1200
minutes, but it was running fine, completed all HDFS and MAPREDUCED tests, and
most of the YARN tests. Given more time I think it would have succeeded.
It is still not clear why trunk succeeds with this patch in. But if we revert
it from branch-2 we may at least be able to restore the pre-commit builds on
it. What people think?
I propose to attach a patch reverting HADOOP-15251 to this jira and try the
pre-commit build.
was (Author: shv):
[Ran a build for
branch-2.7|https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/11/],
which doesn't have HADOOP-13514 backport. It was aborted after 1200 minutes,
but it was running fine, completed all HDFS and MAPREDUCED tests, and most of
the YARN tests. Given more time I think it would have succeeded.
It is still not clear what why trunk succeeds with this patch in. But if we
revert it from brach-2 we may at least be able to restore the pre-commit builds
on it. What people think?
I propose to attach a patch reverting HADOOP-15251 to this jira and try the
pre-commit build.
> Fix branch-2 builds
> ---
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Jonathan Hung
>Priority: Critical
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15711) Fix branch-2 builds
[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621256#comment-16621256
]
Konstantin Shvachko commented on HADOOP-15711:
--
[Ran a build for
branch-2.7|https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/11/],
which doesn't have HADOOP-13514 backport. It was aborted after 1200 minutes,
but it was running fine, completed all HDFS and MAPREDUCED tests, and most of
the YARN tests. Given more time I think it would have succeeded.
It is still not clear what why trunk succeeds with this patch in. But if we
revert it from brach-2 we may at least be able to restore the pre-commit builds
on it. What people think?
I propose to attach a patch reverting HADOOP-15251 to this jira and try the
pre-commit build.
> Fix branch-2 builds
> ---
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Jonathan Hung
>Priority: Critical
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15726) Create utility to limit frequency of log statements
[
https://issues.apache.org/jira/browse/HADOOP-15726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16619972#comment-16619972
]
Konstantin Shvachko commented on HADOOP-15726:
--
v002 look great.
> Create utility to limit frequency of log statements
> ---
>
> Key: HADOOP-15726
> URL: https://issues.apache.org/jira/browse/HADOOP-15726
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, util
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HADOOP-15726.000.patch, HADOOP-15726.001.patch,
> HADOOP-15726.002.patch
>
>
> There is a common pattern of logging a behavior that is normally extraneous.
> Under some circumstances, such a behavior becomes common, flooding the logs
> and making it difficult to see what else is going on in the system. Under
> such situations it is beneficial to limit how frequently the extraneous
> behavior is logged, while capturing some summary information about the
> suppressed log statements.
> This is currently implemented in {{FSNamesystemLock}} (in HDFS-10713). We
> have additional use cases for this in HDFS-13791, so this is a good time to
> create a common utility for different sites to share this logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15711) Fix branch-2 builds
[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609741#comment-16609741
]
Konstantin Shvachko commented on HADOOP-15711:
--
I suggest we do two things:
# Isolate a commit which caused stopping sending the emails when the build
finishes.
# Identify issues that improved running tests on trunk compared to branch-2.
Alternatively we can just switch to old-fashioned test-patch build in favor of
yetus. It seems that our local Jenkins with test-patch based build is
succeeding on branch-2, that is, some tests are failing, but the build does not
timeout.
> Fix branch-2 builds
> ---
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Jonathan Hung
>Priority: Critical
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14891) Remove references to Guava Objects.toStringHelper
[
https://issues.apache.org/jira/browse/HADOOP-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520797#comment-16520797
]
Konstantin Shvachko commented on HADOOP-14891:
--
+1 Makes sense to backport to branch-2.7.
Seems to be applying cleanly.
If I were looking at it from the start I would've suggested to use the original
chain call pattern, like {{sb.append(..).append(..)}}.
May be something for the future.
> Remove references to Guava Objects.toStringHelper
> -
>
> Key: HADOOP-14891
> URL: https://issues.apache.org/jira/browse/HADOOP-14891
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.0, 2.8.1, 2.7.8
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Fix For: 2.9.0, 2.8.3
>
> Attachments: HADOOP-14891.001-branch-2.patch
>
>
> Use provided a guava 23.0 jar as part of the job submission.
> {code}
> 2017-09-20 16:10:42,897 [INFO] [main] |service.AbstractService|: Service
> org.apache.tez.dag.app.DAGAppMaster failed in state STARTED; cause:
> org.apache.hadoop.service.ServiceStateException: java.lang.NoSuchMethodError:
> com.google.common.base.Objects.toStringHelper(Ljava/lang/Object;)Lcom/google/common/base/Objects$ToStringHelper;
> org.apache.hadoop.service.ServiceStateException: java.lang.NoSuchMethodError:
> com.google.common.base.Objects.toStringHelper(Ljava/lang/Object;)Lcom/google/common/base/Objects$ToStringHelper;
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
> at
> org.apache.tez.dag.app.DAGAppMaster.startServices(DAGAppMaster.java:1989)
> at
> org.apache.tez.dag.app.DAGAppMaster.serviceStart(DAGAppMaster.java:2056)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2707)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1936)
> at
> org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2703)
> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2508)
> Caused by: java.lang.NoSuchMethodError:
> com.google.common.base.Objects.toStringHelper(Ljava/lang/Object;)Lcom/google/common/base/Objects$ToStringHelper;
> at
> org.apache.hadoop.metrics2.lib.MetricsRegistry.toString(MetricsRegistry.java:419)
> at java.lang.String.valueOf(String.java:2994)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at org.apache.hadoop.ipc.metrics.RpcMetrics.(RpcMetrics.java:74)
> at org.apache.hadoop.ipc.metrics.RpcMetrics.create(RpcMetrics.java:80)
> at org.apache.hadoop.ipc.Server.(Server.java:2658)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:968)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:367)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:342)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:810)
> at
> org.apache.tez.dag.api.client.DAGClientServer.createServer(DAGClientServer.java:134)
> at
> org.apache.tez.dag.api.client.DAGClientServer.serviceStart(DAGClientServer.java:82)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.tez.dag.app.DAGAppMaster$ServiceWithDependency.start(DAGAppMaster.java:1909)
> at
> org.apache.tez.dag.app.DAGAppMaster$ServiceThread.run(DAGAppMaster.java:1930)
> 2017-09-20 16:10:42,898 [ERROR] [main] |rm.TaskSchedulerManager|: Failed to
> do a clean initiateStop for Scheduler: [0:TezYarn]
> {code}
> Metrics2 has been relying on deprecated toStringHelper for some time now
> which was finally removed in guava 21.0. Removing the dependency on this
> method will free up the user to supplying their own guava jar again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15253) Should update maxQueueSize when refresh call queue
[
https://issues.apache.org/jira/browse/HADOOP-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441449#comment-16441449
]
Konstantin Shvachko commented on HADOOP-15253:
--
Hey [~Tao Jie] any progress on branch-2.7 patch? Or should we close this?
> Should update maxQueueSize when refresh call queue
> --
>
> Key: HADOOP-15253
> URL: https://issues.apache.org/jira/browse/HADOOP-15253
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.8.2
>Reporter: Tao Jie
>Assignee: Tao Jie
>Priority: Minor
> Attachments: HADOOP-15253.001.patch, HADOOP-15253.002.patch,
> HADOOP-15253.003.patch, HADOOP-15253.004.patch
>
>
> When calling {{dfsadmin -refreshCallQueue}} to update CallQueue instance,
> {{maxQueueSize}} should also be updated.
> In case of changing CallQueue instance to FairCallQueue, the length of each
> queue in FairCallQueue would be 1/priorityLevels of original length of
> DefaultCallQueue. So it would be helpful for us to set the length of
> callqueue to a proper value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15205) maven release: missing source attachments for hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/HADOOP-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440246#comment-16440246 ] Konstantin Shvachko commented on HADOOP-15205: -- Thanks [~eddyxu] for looking into this. I suggest we update the how-to-release page for now, but we should fix it so that "mvn deploy -Psign -DskipTests" worked. I am definitely not qualified to deal with maven builds. Do you think you can track this, Eddy? > maven release: missing source attachments for hadoop-mapreduce-client-core > -- > > Key: HADOOP-15205 > URL: https://issues.apache.org/jira/browse/HADOOP-15205 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.5, 3.0.0 >Reporter: Zoltan Haindrich >Priority: Major > > I wanted to use the source attachment; however it looks like since 2.7.5 that > artifact is not present at maven central ; it looks like the last release > which had source attachments / javadocs was 2.7.4 > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.5/ > this seems to be not limited to mapreduce; as the same change is present for > yarn-common as well > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.5/ > and also hadoop-common > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.5/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.0.0/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14970) MiniHadoopClusterManager doesn't respect lack of format option
[
https://issues.apache.org/jira/browse/HADOOP-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-14970:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.3
2.9.2
3.1.1
3.2.0
2.8.4
2.10.0
2.7.7
Status: Resolved (was: Patch Available)
I just committed this down to branch-2.7. Thank you [~xkrogen].
> MiniHadoopClusterManager doesn't respect lack of format option
> --
>
> Key: HADOOP-14970
> URL: https://issues.apache.org/jira/browse/HADOOP-14970
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 2.7.7, 2.10.0, 2.8.4, 3.2.0, 3.1.1, 2.9.2, 3.0.3
>
> Attachments: HADOOP-14970.000.patch
>
>
> The CLI MiniCluster, {{MiniHadoopClusterManager}}, says that by default it
> does not format its directories, and provides the {{-format}} option to
> specify that it should do so. However, it builds its {{MiniDFSCluster}} like:
> {code}
> dfs = new MiniDFSCluster.Builder(conf).nameNodePort(nnPort)
> .nameNodeHttpPort(nnHttpPort).numDataNodes(numDataNodes)
> .startupOption(dfsOpts).build();
> {code}
> {{MiniDFSCluster.Builder}}, by default, sets {{format}} to true, so even
> though the {{startupOption}} is {{REGULAR}}, it will still format regardless
> of whether or not the flag is supplied.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14970) MiniHadoopClusterManager doesn't respect lack of format option
[
https://issues.apache.org/jira/browse/HADOOP-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436028#comment-16436028
]
Konstantin Shvachko commented on HADOOP-14970:
--
+1 on the change. Should not format DNs if the {{StartupOption.REGULAR}}. Will
commit in a bit.
> MiniHadoopClusterManager doesn't respect lack of format option
> --
>
> Key: HADOOP-14970
> URL: https://issues.apache.org/jira/browse/HADOOP-14970
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: HADOOP-14970.000.patch
>
>
> The CLI MiniCluster, {{MiniHadoopClusterManager}}, says that by default it
> does not format its directories, and provides the {{-format}} option to
> specify that it should do so. However, it builds its {{MiniDFSCluster}} like:
> {code}
> dfs = new MiniDFSCluster.Builder(conf).nameNodePort(nnPort)
> .nameNodeHttpPort(nnHttpPort).numDataNodes(numDataNodes)
> .startupOption(dfsOpts).build();
> {code}
> {{MiniDFSCluster.Builder}}, by default, sets {{format}} to true, so even
> though the {{startupOption}} is {{REGULAR}}, it will still format regardless
> of whether or not the flag is supplied.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13672) Extract out jackson calls into an overrideable method in DelegationTokenAuthenticationHandler
[ https://issues.apache.org/jira/browse/HADOOP-13672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HADOOP-13672: - Target Version/s: (was: 2.7.6) Status: Open (was: Patch Available) The patch got stale, canceling. > Extract out jackson calls into an overrideable method in > DelegationTokenAuthenticationHandler > - > > Key: HADOOP-13672 > URL: https://issues.apache.org/jira/browse/HADOOP-13672 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Minor > Attachments: HADOOP-13672.patch, HADOOP-13672.patch > > > In Apache Solr, we use hadoop-auth for delegation tokens. However, because of > the following lines, we need to import Jackson (old version). > https://github.com/apache/hadoop/blob/branch-2.7/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java#L279 > If we could extract out the calls to ObjectMapper to another method, so that > at Solr we could override it to do the Map -> json conversion using noggit, > it would be helpful. > Reference: SOLR-9542 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14689) PreCommit-HADOOP-Build on branch-2.7 is failing.
[ https://issues.apache.org/jira/browse/HADOOP-14689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HADOOP-14689: - Resolution: Won't Fix Status: Resolved (was: Patch Available) > PreCommit-HADOOP-Build on branch-2.7 is failing. > > > Key: HADOOP-14689 > URL: https://issues.apache.org/jira/browse/HADOOP-14689 > Project: Hadoop Common > Issue Type: Task >Reporter: Konstantin Shvachko >Assignee: Konstantin Shvachko >Priority: Major > Labels: build > Attachments: HADOOP-14689-branch-2.7.001.patch > > > Will add dummy patches to test Jenkins PreCommit-HADOOP-Build failures on > branch-2.7. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14989) metrics2 JMX cache refresh result in inconsistent Mutable(Stat|Rate) values
[
https://issues.apache.org/jira/browse/HADOOP-14989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko reassigned HADOOP-14989:
Assignee: (was: Erik Krogen)
> metrics2 JMX cache refresh result in inconsistent Mutable(Stat|Rate) values
> ---
>
> Key: HADOOP-14989
> URL: https://issues.apache.org/jira/browse/HADOOP-14989
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.6.5
>Reporter: Erik Krogen
>Priority: Critical
> Attachments: HADOOP-14989.test.patch
>
>
> While doing some digging in the metrics2 system recently, we noticed that the
> way {{MutableStat}} values are collected (and thus {{MutableRate}}, since it
> is based off of {{MutableStat}}) mean that every time the value is
> snapshotted, all previous information is lost. So every time a JMX cache
> refresh occurs, it resets the {{MutableStat}}, meaning that all configured
> metrics sinks do not consider the previous statistics in their emitted
> values. The same behavior is true if you configured multiple sink periods.
> {{MutableStat}}, to compute its average value, maintains a total value since
> last snapshot, as well as operation count since last snapshot. Upon
> snapshotting, the average is calculated as (total / opCount) and placed into
> a gauge metric, and total / operation count are cleared. So the average value
> represents the average since the last snapshot. If we have only a single sink
> period ever snapshotting, this would result in the expected behavior that the
> value is the average over the reporting period. However, if multiple sink
> periods are configured, or if the JMX cache is refreshed, this is another
> snapshot operation. So, for example, if you have a FileSink configured at a
> 60 second interval and your JMX cache refreshes itself 1 second before the
> FileSink period fires, the values emitted to your FileSink only represent
> averages _over the last one second_.
> A few ways to solve this issue:
> * Make {{MutableRate}} manage its own average refresh, similar to
> {{MutableQuantiles}}, which has a refresh thread and saves a snapshot of the
> last quantile values that it will serve up until the next refresh. Given how
> many {{MutableRate}} metrics there are, a thread per metric is not really
> feasible, but could be done on e.g. a per-source basis. This has some
> downsides: if multiple sinks are configured with different periods, what is
> the right refresh period for the {{MutableRate}}?
> * Make {{MutableRate}} emit two counters, one for total and one for operation
> count, rather than an average gauge and an operation count counter. The
> average could then be calculated downstream from this information. This is
> cumbersome for operators and not backwards compatible. To improve on both of
> those downsides, we could have it keep the current behavior but
> _additionally_ emit the total as a counter. The snapshotted average is
> probably sufficient in the common case (we've been using it for years), and
> when more guaranteed accuracy is required, the average could be derived from
> the total and operation count.
> The two above suggestions will fix this for both JMX and multiple sink
> periods, but may be overkill. Multiple sink periods are probably not
> necessary though we should at least document the behavior.
> Open to suggestions & input here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15253) Should update maxQueueSize when refresh call queue
[
https://issues.apache.org/jira/browse/HADOOP-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421157#comment-16421157
]
Konstantin Shvachko commented on HADOOP-15253:
--
I just committed this to all branches up to branch-2.8. Thanks you [~Tao Jie].
If we want this in branch-2.7 we will need a separate patch. Let me know if we
do.
> Should update maxQueueSize when refresh call queue
> --
>
> Key: HADOOP-15253
> URL: https://issues.apache.org/jira/browse/HADOOP-15253
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.8.2
>Reporter: Tao Jie
>Assignee: Tao Jie
>Priority: Minor
> Attachments: HADOOP-15253.001.patch, HADOOP-15253.002.patch,
> HADOOP-15253.003.patch, HADOOP-15253.004.patch
>
>
> When calling {{dfsadmin -refreshCallQueue}} to update CallQueue instance,
> {{maxQueueSize}} should also be updated.
> In case of changing CallQueue instance to FairCallQueue, the length of each
> queue in FairCallQueue would be 1/priorityLevels of original length of
> DefaultCallQueue. So it would be helpful for us to set the length of
> callqueue to a proper value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12862) LDAP Group Mapping over SSL can not specify trust store
[
https://issues.apache.org/jira/browse/HADOOP-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-12862:
-
Labels: (was: release-blocker)
> LDAP Group Mapping over SSL can not specify trust store
> ---
>
> Key: HADOOP-12862
> URL: https://issues.apache.org/jira/browse/HADOOP-12862
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 2.10.0, 2.9.1, 2.8.4, 2.7.6, 3.2.0, 3.0.2, 3.1.1
>
> Attachments: HADOOP-12862.001.patch, HADOOP-12862.002.patch,
> HADOOP-12862.003.patch, HADOOP-12862.004.patch, HADOOP-12862.005.patch,
> HADOOP-12862.006.patch, HADOOP-12862.007.patch, HADOOP-12862.008.patch,
> HADOOP-12862.009.patch
>
>
> In a secure environment, SSL is used to encrypt LDAP request for group
> mapping resolution.
> We (+[~yoderme], +[~tgrayson]) have found that its implementation is strange.
> For information, Hadoop name node, as an LDAP client, talks to a LDAP server
> to resolve the group mapping of a user. In the case of LDAP over SSL, a
> typical scenario is to establish one-way authentication (the client verifies
> the server's certificate is real) by storing the server's certificate in the
> client's truststore.
> A rarer scenario is to establish two-way authentication: in addition to store
> truststore for the client to verify the server, the server also verifies the
> client's certificate is real, and the client stores its own certificate in
> its keystore.
> However, the current implementation for LDAP over SSL does not seem to be
> correct in that it only configures keystore but no truststore (so LDAP server
> can verify Hadoop's certificate, but Hadoop may not be able to verify LDAP
> server's certificate)
> I think there should an extra pair of properties to specify the
> truststore/password for LDAP server, and use that to configure system
> properties {{javax.net.ssl.trustStore}}/{{javax.net.ssl.trustStorePassword}}
> I am a security layman so my words can be imprecise. But I hope this makes
> sense.
> Oracle's SSL LDAP documentation:
> http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ssl.html
> JSSE reference guide:
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15355) TestCommonConfigurationFields is broken by HADOOP-15312
[ https://issues.apache.org/jira/browse/HADOOP-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420759#comment-16420759 ] Konstantin Shvachko commented on HADOOP-15355: -- +1 Looks good, fixes the test. > TestCommonConfigurationFields is broken by HADOOP-15312 > --- > > Key: HADOOP-15355 > URL: https://issues.apache.org/jira/browse/HADOOP-15355 > Project: Hadoop Common > Issue Type: Bug > Components: test >Affects Versions: 2.10.0 >Reporter: Konstantin Shvachko >Assignee: LiXin Ge >Priority: Major > Attachments: HADOOP-15355.001.patch, HADOOP-15355.002.patch > > > TestCommonConfigurationFields is failing after HADOOP-15312. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15312) Undocumented KeyProvider configuration keys
[ https://issues.apache.org/jira/browse/HADOOP-15312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420174#comment-16420174 ] Konstantin Shvachko commented on HADOOP-15312: -- Hey guys let's target fixing the test under HADOOP-15355. > Undocumented KeyProvider configuration keys > --- > > Key: HADOOP-15312 > URL: https://issues.apache.org/jira/browse/HADOOP-15312 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: LiXin Ge >Priority: Major > Fix For: 3.1.0, 2.10.0, 3.0.2 > > Attachments: HADOOP-15312.001.patch, HADOOP-15312.002.patch, > HADOOP-15312.003.patch > > > Via HADOOP-14445, I found two undocumented configuration keys: > hadoop.security.key.default.bitlength and hadoop.security.key.default.cipher -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-15355) TestCommonConfigurationFields is broken by HADOOP-15312
[ https://issues.apache.org/jira/browse/HADOOP-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko moved HDFS-13374 to HADOOP-15355: - Affects Version/s: (was: 2.10.0) 2.10.0 Target Version/s: 2.10.0 (was: 2.10.0) Component/s: (was: test) test Key: HADOOP-15355 (was: HDFS-13374) Project: Hadoop Common (was: Hadoop HDFS) > TestCommonConfigurationFields is broken by HADOOP-15312 > --- > > Key: HADOOP-15355 > URL: https://issues.apache.org/jira/browse/HADOOP-15355 > Project: Hadoop Common > Issue Type: Bug > Components: test >Affects Versions: 2.10.0 >Reporter: Konstantin Shvachko >Priority: Major > > TestCommonConfigurationFields is failing after HADOOP-15312. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15325) Make Configuration#getPasswordFromCredentialsProvider() a public API
[ https://issues.apache.org/jira/browse/HADOOP-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420039#comment-16420039 ] Konstantin Shvachko commented on HADOOP-15325: -- Hey [~ste...@apache.org] there is another jira HDFS-13366 to deprecate password fields. Could you please explain there the google cloud service use case in more details. I thought we should not store plain-text passwords in config files ever since it is not secure, but may be I missed some cases. > Make Configuration#getPasswordFromCredentialsProvider() a public API > > > Key: HADOOP-15325 > URL: https://issues.apache.org/jira/browse/HADOOP-15325 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 2.6.0 >Reporter: Wei-Chiu Chuang >Assignee: Zsolt Venczel >Priority: Major > > HADOOP-10607 added a public API Configuration.getPassword() which reads > passwords from credential provider and then falls back to reading from > configuration if one is not available. > This API has been used throughout Hadoop codebase and downstream > applications. It is understandable for old password configuration keys to > fallback to configuration to maintain backward compatibility. But for new > configuration passwords that don't have legacy, there should be an option to > _not_ fallback, because storing passwords in configuration is considered a > bad security practice. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15325) Make Configuration#getPasswordFromCredentialsProvider() a public API
[ https://issues.apache.org/jira/browse/HADOOP-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko resolved HADOOP-15325. -- Resolution: Implemented This has been incorporated into HADOOP-12862. > Make Configuration#getPasswordFromCredentialsProvider() a public API > > > Key: HADOOP-15325 > URL: https://issues.apache.org/jira/browse/HADOOP-15325 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 2.6.0 >Reporter: Wei-Chiu Chuang >Assignee: Zsolt Venczel >Priority: Major > > HADOOP-10607 added a public API Configuration.getPassword() which reads > passwords from credential provider and then falls back to reading from > configuration if one is not available. > This API has been used throughout Hadoop codebase and downstream > applications. It is understandable for old password configuration keys to > fallback to configuration to maintain backward compatibility. But for new > configuration passwords that don't have legacy, there should be an option to > _not_ fallback, because storing passwords in configuration is considered a > bad security practice. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12862) LDAP Group Mapping over SSL can not specify trust store
[
https://issues.apache.org/jira/browse/HADOOP-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-12862:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.1
3.0.2
3.2.0
2.7.6
2.8.4
2.9.1
2.10.0
Status: Resolved (was: Patch Available)
I just committed this to the following branches:
{code}
2c6cfad5a31..2216bde3229 trunk -> trunk
2960592c6f9..99b5b9dce1b branch-3.1 -> branch-3.1
cd74a281a76..3912bdb2b03 branch-3.0 -> branch-3.0
7e5c8faeb7c..c138682e048 branch-2 -> branch-2
f71128f056f..5cbbc4c374b branch-2.9 -> branch-2.9
dff546156dc..fb15e41737d branch-2.8 -> branch-2.8
65baefb5381..caf518cd3f1 branch-2.7 -> branch-2.7
{code}
Thank you [~jojochuang].
> LDAP Group Mapping over SSL can not specify trust store
> ---
>
> Key: HADOOP-12862
> URL: https://issues.apache.org/jira/browse/HADOOP-12862
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Labels: release-blocker
> Fix For: 2.10.0, 2.9.1, 2.8.4, 2.7.6, 3.2.0, 3.0.2, 3.1.1
>
> Attachments: HADOOP-12862.001.patch, HADOOP-12862.002.patch,
> HADOOP-12862.003.patch, HADOOP-12862.004.patch, HADOOP-12862.005.patch,
> HADOOP-12862.006.patch, HADOOP-12862.007.patch, HADOOP-12862.008.patch,
> HADOOP-12862.009.patch
>
>
> In a secure environment, SSL is used to encrypt LDAP request for group
> mapping resolution.
> We (+[~yoderme], +[~tgrayson]) have found that its implementation is strange.
> For information, Hadoop name node, as an LDAP client, talks to a LDAP server
> to resolve the group mapping of a user. In the case of LDAP over SSL, a
> typical scenario is to establish one-way authentication (the client verifies
> the server's certificate is real) by storing the server's certificate in the
> client's truststore.
> A rarer scenario is to establish two-way authentication: in addition to store
> truststore for the client to verify the server, the server also verifies the
> client's certificate is real, and the client stores its own certificate in
> its keystore.
> However, the current implementation for LDAP over SSL does not seem to be
> correct in that it only configures keystore but no truststore (so LDAP server
> can verify Hadoop's certificate, but Hadoop may not be able to verify LDAP
> server's certificate)
> I think there should an extra pair of properties to specify the
> truststore/password for LDAP server, and use that to configure system
> properties {{javax.net.ssl.trustStore}}/{{javax.net.ssl.trustStorePassword}}
> I am a security layman so my words can be imprecise. But I hope this makes
> sense.
> Oracle's SSL LDAP documentation:
> http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ssl.html
> JSSE reference guide:
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12862) LDAP Group Mapping over SSL can not specify trust store
[
https://issues.apache.org/jira/browse/HADOOP-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419982#comment-16419982
]
Konstantin Shvachko commented on HADOOP-12862:
--
The test failure seems to be introduced by HADOOP-15312.
Thanks for the review [~zhz], will commit now.
> LDAP Group Mapping over SSL can not specify trust store
> ---
>
> Key: HADOOP-12862
> URL: https://issues.apache.org/jira/browse/HADOOP-12862
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Labels: release-blocker
> Attachments: HADOOP-12862.001.patch, HADOOP-12862.002.patch,
> HADOOP-12862.003.patch, HADOOP-12862.004.patch, HADOOP-12862.005.patch,
> HADOOP-12862.006.patch, HADOOP-12862.007.patch, HADOOP-12862.008.patch,
> HADOOP-12862.009.patch
>
>
> In a secure environment, SSL is used to encrypt LDAP request for group
> mapping resolution.
> We (+[~yoderme], +[~tgrayson]) have found that its implementation is strange.
> For information, Hadoop name node, as an LDAP client, talks to a LDAP server
> to resolve the group mapping of a user. In the case of LDAP over SSL, a
> typical scenario is to establish one-way authentication (the client verifies
> the server's certificate is real) by storing the server's certificate in the
> client's truststore.
> A rarer scenario is to establish two-way authentication: in addition to store
> truststore for the client to verify the server, the server also verifies the
> client's certificate is real, and the client stores its own certificate in
> its keystore.
> However, the current implementation for LDAP over SSL does not seem to be
> correct in that it only configures keystore but no truststore (so LDAP server
> can verify Hadoop's certificate, but Hadoop may not be able to verify LDAP
> server's certificate)
> I think there should an extra pair of properties to specify the
> truststore/password for LDAP server, and use that to configure system
> properties {{javax.net.ssl.trustStore}}/{{javax.net.ssl.trustStorePassword}}
> I am a security layman so my words can be imprecise. But I hope this makes
> sense.
> Oracle's SSL LDAP documentation:
> http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ssl.html
> JSSE reference guide:
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15312) Undocumented KeyProvider configuration keys
[
https://issues.apache.org/jira/browse/HADOOP-15312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419954#comment-16419954
]
Konstantin Shvachko commented on HADOOP-15312:
--
This seems broke the test {{TestCommonConfigurationFields}}, which is reported
by Jenkins here.
> Undocumented KeyProvider configuration keys
> ---
>
> Key: HADOOP-15312
> URL: https://issues.apache.org/jira/browse/HADOOP-15312
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: LiXin Ge
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 3.2.0, 3.0.2
>
> Attachments: HADOOP-15312.001.patch, HADOOP-15312.002.patch
>
>
> Via HADOOP-14445, I found two undocumented configuration keys:
> hadoop.security.key.default.bitlength and hadoop.security.key.default.cipher
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12862) LDAP Group Mapping over SSL can not specify trust store
[
https://issues.apache.org/jira/browse/HADOOP-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418563#comment-16418563
]
Konstantin Shvachko commented on HADOOP-12862:
--
Looks like we all in agreement here. To accelerate things attaching a
modification of the last patch, which
# removes {{hadoop.security.group.mapping.ldap.ssl.truststore.password}} from
{{core-default.xml}}
# makes {{getPasswordFromCredentialProviders()}} public, and
# updates some comments to clarify things
> LDAP Group Mapping over SSL can not specify trust store
> ---
>
> Key: HADOOP-12862
> URL: https://issues.apache.org/jira/browse/HADOOP-12862
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Labels: release-blocker
> Attachments: HADOOP-12862.001.patch, HADOOP-12862.002.patch,
> HADOOP-12862.003.patch, HADOOP-12862.004.patch, HADOOP-12862.005.patch,
> HADOOP-12862.006.patch, HADOOP-12862.007.patch, HADOOP-12862.008.patch,
> HADOOP-12862.009.patch
>
>
> In a secure environment, SSL is used to encrypt LDAP request for group
> mapping resolution.
> We (+[~yoderme], +[~tgrayson]) have found that its implementation is strange.
> For information, Hadoop name node, as an LDAP client, talks to a LDAP server
> to resolve the group mapping of a user. In the case of LDAP over SSL, a
> typical scenario is to establish one-way authentication (the client verifies
> the server's certificate is real) by storing the server's certificate in the
> client's truststore.
> A rarer scenario is to establish two-way authentication: in addition to store
> truststore for the client to verify the server, the server also verifies the
> client's certificate is real, and the client stores its own certificate in
> its keystore.
> However, the current implementation for LDAP over SSL does not seem to be
> correct in that it only configures keystore but no truststore (so LDAP server
> can verify Hadoop's certificate, but Hadoop may not be able to verify LDAP
> server's certificate)
> I think there should an extra pair of properties to specify the
> truststore/password for LDAP server, and use that to configure system
> properties {{javax.net.ssl.trustStore}}/{{javax.net.ssl.trustStorePassword}}
> I am a security layman so my words can be imprecise. But I hope this makes
> sense.
> Oracle's SSL LDAP documentation:
> http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ssl.html
> JSSE reference guide:
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12862) LDAP Group Mapping over SSL can not specify trust store
[
https://issues.apache.org/jira/browse/HADOOP-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-12862:
-
Attachment: HADOOP-12862.009.patch
> LDAP Group Mapping over SSL can not specify trust store
> ---
>
> Key: HADOOP-12862
> URL: https://issues.apache.org/jira/browse/HADOOP-12862
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Labels: release-blocker
> Attachments: HADOOP-12862.001.patch, HADOOP-12862.002.patch,
> HADOOP-12862.003.patch, HADOOP-12862.004.patch, HADOOP-12862.005.patch,
> HADOOP-12862.006.patch, HADOOP-12862.007.patch, HADOOP-12862.008.patch,
> HADOOP-12862.009.patch
>
>
> In a secure environment, SSL is used to encrypt LDAP request for group
> mapping resolution.
> We (+[~yoderme], +[~tgrayson]) have found that its implementation is strange.
> For information, Hadoop name node, as an LDAP client, talks to a LDAP server
> to resolve the group mapping of a user. In the case of LDAP over SSL, a
> typical scenario is to establish one-way authentication (the client verifies
> the server's certificate is real) by storing the server's certificate in the
> client's truststore.
> A rarer scenario is to establish two-way authentication: in addition to store
> truststore for the client to verify the server, the server also verifies the
> client's certificate is real, and the client stores its own certificate in
> its keystore.
> However, the current implementation for LDAP over SSL does not seem to be
> correct in that it only configures keystore but no truststore (so LDAP server
> can verify Hadoop's certificate, but Hadoop may not be able to verify LDAP
> server's certificate)
> I think there should an extra pair of properties to specify the
> truststore/password for LDAP server, and use that to configure system
> properties {{javax.net.ssl.trustStore}}/{{javax.net.ssl.trustStorePassword}}
> I am a security layman so my words can be imprecise. But I hope this makes
> sense.
> Oracle's SSL LDAP documentation:
> http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ssl.html
> JSSE reference guide:
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15325) Add an option to make Configuration.getPassword() not to fallback to read passwords from configuration.
[
https://issues.apache.org/jira/browse/HADOOP-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416122#comment-16416122
]
Konstantin Shvachko commented on HADOOP-15325:
--
??I think that you are viewing the use of getPassword as a mechanism only to be
used for old passwords that used to be stored in configuration and that any new
ones should use the credential provider API directly instead.??
Hi [~lmccay], you are exactly right. And even though {{getPassowrd()}} is nice
and convenient, we need some way to introduce new practice, which avoids adding
passwords into configs. So {{public getPasswordFromCredentialsProvider()}} as
you and [~jojochuang] suggested should serve this purpose well.
I am not in favor of adding the extra FALLBACK parameter because it is quite
confusing / complicates configuration. I mean you really have to look in the
code to understand what it means. And potentially read this discussion to
understand the background.
I also think we should deprecate all existing password fields in current
configuration and backport it into downstream versions, so that we could remove
them some time in the future.
> Add an option to make Configuration.getPassword() not to fallback to read
> passwords from configuration.
> ---
>
> Key: HADOOP-15325
> URL: https://issues.apache.org/jira/browse/HADOOP-15325
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Zsolt Venczel
>Priority: Major
>
> HADOOP-10607 added a public API Configuration.getPassword() which reads
> passwords from credential provider and then falls back to reading from
> configuration if one is not available.
> This API has been used throughout Hadoop codebase and downstream
> applications. It is understandable for old password configuration keys to
> fallback to configuration to maintain backward compatibility. But for new
> configuration passwords that don't have legacy, there should be an option to
> _not_ fallback, because storing passwords in configuration is considered a
> bad security practice.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12862) LDAP Group Mapping over SSL can not specify trust store
[
https://issues.apache.org/jira/browse/HADOOP-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-12862:
-
Labels: release-blocker (was: )
Target Version/s: 2.7.6
Marked as blocker for 2.7.6, because the feature doesn't work without this.
> LDAP Group Mapping over SSL can not specify trust store
> ---
>
> Key: HADOOP-12862
> URL: https://issues.apache.org/jira/browse/HADOOP-12862
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Labels: release-blocker
> Attachments: HADOOP-12862.001.patch, HADOOP-12862.002.patch,
> HADOOP-12862.003.patch, HADOOP-12862.004.patch, HADOOP-12862.005.patch,
> HADOOP-12862.006.patch, HADOOP-12862.007.patch, HADOOP-12862.008.patch
>
>
> In a secure environment, SSL is used to encrypt LDAP request for group
> mapping resolution.
> We (+[~yoderme], +[~tgrayson]) have found that its implementation is strange.
> For information, Hadoop name node, as an LDAP client, talks to a LDAP server
> to resolve the group mapping of a user. In the case of LDAP over SSL, a
> typical scenario is to establish one-way authentication (the client verifies
> the server's certificate is real) by storing the server's certificate in the
> client's truststore.
> A rarer scenario is to establish two-way authentication: in addition to store
> truststore for the client to verify the server, the server also verifies the
> client's certificate is real, and the client stores its own certificate in
> its keystore.
> However, the current implementation for LDAP over SSL does not seem to be
> correct in that it only configures keystore but no truststore (so LDAP server
> can verify Hadoop's certificate, but Hadoop may not be able to verify LDAP
> server's certificate)
> I think there should an extra pair of properties to specify the
> truststore/password for LDAP server, and use that to configure system
> properties {{javax.net.ssl.trustStore}}/{{javax.net.ssl.trustStorePassword}}
> I am a security layman so my words can be imprecise. But I hope this makes
> sense.
> Oracle's SSL LDAP documentation:
> http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ssl.html
> JSSE reference guide:
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12862) LDAP Group Mapping over SSL can not specify trust store
[
https://issues.apache.org/jira/browse/HADOOP-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414765#comment-16414765
]
Konstantin Shvachko commented on HADOOP-12862:
--
I see what you mean now. The same name
{{hadoop.security.group.mapping.ldap.ssl.truststore.password}} used both as a
config variable and as a key in a {{CredentialProvider}}. We want to remove the
former, but keep the latter.
I think making {{Configuration.getPasswordFromCredentialProviders()}} public is
the right direction. Then we explicitly will not fall back to reading from
configuration for truststore.
> LDAP Group Mapping over SSL can not specify trust store
> ---
>
> Key: HADOOP-12862
> URL: https://issues.apache.org/jira/browse/HADOOP-12862
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HADOOP-12862.001.patch, HADOOP-12862.002.patch,
> HADOOP-12862.003.patch, HADOOP-12862.004.patch, HADOOP-12862.005.patch,
> HADOOP-12862.006.patch, HADOOP-12862.007.patch, HADOOP-12862.008.patch
>
>
> In a secure environment, SSL is used to encrypt LDAP request for group
> mapping resolution.
> We (+[~yoderme], +[~tgrayson]) have found that its implementation is strange.
> For information, Hadoop name node, as an LDAP client, talks to a LDAP server
> to resolve the group mapping of a user. In the case of LDAP over SSL, a
> typical scenario is to establish one-way authentication (the client verifies
> the server's certificate is real) by storing the server's certificate in the
> client's truststore.
> A rarer scenario is to establish two-way authentication: in addition to store
> truststore for the client to verify the server, the server also verifies the
> client's certificate is real, and the client stores its own certificate in
> its keystore.
> However, the current implementation for LDAP over SSL does not seem to be
> correct in that it only configures keystore but no truststore (so LDAP server
> can verify Hadoop's certificate, but Hadoop may not be able to verify LDAP
> server's certificate)
> I think there should an extra pair of properties to specify the
> truststore/password for LDAP server, and use that to configure system
> properties {{javax.net.ssl.trustStore}}/{{javax.net.ssl.trustStorePassword}}
> I am a security layman so my words can be imprecise. But I hope this makes
> sense.
> Oracle's SSL LDAP documentation:
> http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ssl.html
> JSSE reference guide:
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15325) Add an option to make Configuration.getPassword() not to fallback to read passwords from configuration.
[ https://issues.apache.org/jira/browse/HADOOP-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414568#comment-16414568 ] Konstantin Shvachko commented on HADOOP-15325: -- My comment from HADOOP-12862. I don't think this makes sense. It is like adding an optional option to ignore an optional parameter. People should just NOT put passwords in configs. We tolerate previously introduced password parameters for backward compatibility. But we should not add new password fields into configs. > Add an option to make Configuration.getPassword() not to fallback to read > passwords from configuration. > --- > > Key: HADOOP-15325 > URL: https://issues.apache.org/jira/browse/HADOOP-15325 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 2.6.0 >Reporter: Wei-Chiu Chuang >Assignee: Zsolt Venczel >Priority: Major > > HADOOP-10607 added a public API Configuration.getPassword() which reads > passwords from credential provider and then falls back to reading from > configuration if one is not available. > This API has been used throughout Hadoop codebase and downstream > applications. It is understandable for old password configuration keys to > fallback to configuration to maintain backward compatibility. But for new > configuration passwords that don't have legacy, there should be an option to > _not_ fallback, because storing passwords in configuration is considered a > bad security practice. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12862) LDAP Group Mapping over SSL can not specify trust store
[
https://issues.apache.org/jira/browse/HADOOP-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410484#comment-16410484
]
Konstantin Shvachko commented on HADOOP-12862:
--
So the code snippet you cited above was introduced in HADOOP-10607, which
targeted exactly that ??"to eliminate the storage of passwords and secrets in
clear text within configuration files or within code."??
I believe the opportunity to obtain password from configs was left there to
provide backward compatibility.
I think we both agree that storing passwords in config files is a bad idea, no?
So why do we want to keep introducing (optional) password parameters, following
the wrong pattern?
What you propose with HADOOP-15325 is adding an optional option to ignore an
optional parameter. Why?
> LDAP Group Mapping over SSL can not specify trust store
> ---
>
> Key: HADOOP-12862
> URL: https://issues.apache.org/jira/browse/HADOOP-12862
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HADOOP-12862.001.patch, HADOOP-12862.002.patch,
> HADOOP-12862.003.patch, HADOOP-12862.004.patch, HADOOP-12862.005.patch,
> HADOOP-12862.006.patch, HADOOP-12862.007.patch, HADOOP-12862.008.patch
>
>
> In a secure environment, SSL is used to encrypt LDAP request for group
> mapping resolution.
> We (+[~yoderme], +[~tgrayson]) have found that its implementation is strange.
> For information, Hadoop name node, as an LDAP client, talks to a LDAP server
> to resolve the group mapping of a user. In the case of LDAP over SSL, a
> typical scenario is to establish one-way authentication (the client verifies
> the server's certificate is real) by storing the server's certificate in the
> client's truststore.
> A rarer scenario is to establish two-way authentication: in addition to store
> truststore for the client to verify the server, the server also verifies the
> client's certificate is real, and the client stores its own certificate in
> its keystore.
> However, the current implementation for LDAP over SSL does not seem to be
> correct in that it only configures keystore but no truststore (so LDAP server
> can verify Hadoop's certificate, but Hadoop may not be able to verify LDAP
> server's certificate)
> I think there should an extra pair of properties to specify the
> truststore/password for LDAP server, and use that to configure system
> properties {{javax.net.ssl.trustStore}}/{{javax.net.ssl.trustStorePassword}}
> I am a security layman so my words can be imprecise. But I hope this makes
> sense.
> Oracle's SSL LDAP documentation:
> http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ssl.html
> JSSE reference guide:
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12862) LDAP Group Mapping over SSL can not specify trust store
[
https://issues.apache.org/jira/browse/HADOOP-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403075#comment-16403075
]
Konstantin Shvachko edited comment on HADOOP-12862 at 3/22/18 9:40 PM:
---
Sounds like testing is a longer-term issue. BTW if I look into Hadoop
dependencies I see apacheds and ldapsdk. May they be useful for testing. I
wouldn't know.
The testing went well in our environment.
What do you think about removing {{.ssl.truststore.password}}? I really think
people should not use configs for passwords. Typically configs are checked in
in git repositories, so having passwords there is even worse than printing them
on a command line, which [as you
suggested|https://issues.apache.org/jira/browse/HADOOP-15315?focusedCommentId=16399166=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16399166]
is a bad practice.
was (Author: shv):
Sounds like testing is a longer-term issue. BTW if I look into Hadoop
dependencies I see apacheds and ldapsdk. May they be useful for testing. I
wouldn't know.
The testing went well in our environment.
What do you think about removing {{.ssl.truststore.password}}? I really think
people should not use configs for passwords. Typically configs are checked in
in git repositories, so having passwords there is even worth than printing them
on a command line, which [as you
suggested|https://issues.apache.org/jira/browse/HADOOP-15315?focusedCommentId=16399166=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16399166]
is a bad practice.
> LDAP Group Mapping over SSL can not specify trust store
> ---
>
> Key: HADOOP-12862
> URL: https://issues.apache.org/jira/browse/HADOOP-12862
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HADOOP-12862.001.patch, HADOOP-12862.002.patch,
> HADOOP-12862.003.patch, HADOOP-12862.004.patch, HADOOP-12862.005.patch,
> HADOOP-12862.006.patch, HADOOP-12862.007.patch, HADOOP-12862.008.patch
>
>
> In a secure environment, SSL is used to encrypt LDAP request for group
> mapping resolution.
> We (+[~yoderme], +[~tgrayson]) have found that its implementation is strange.
> For information, Hadoop name node, as an LDAP client, talks to a LDAP server
> to resolve the group mapping of a user. In the case of LDAP over SSL, a
> typical scenario is to establish one-way authentication (the client verifies
> the server's certificate is real) by storing the server's certificate in the
> client's truststore.
> A rarer scenario is to establish two-way authentication: in addition to store
> truststore for the client to verify the server, the server also verifies the
> client's certificate is real, and the client stores its own certificate in
> its keystore.
> However, the current implementation for LDAP over SSL does not seem to be
> correct in that it only configures keystore but no truststore (so LDAP server
> can verify Hadoop's certificate, but Hadoop may not be able to verify LDAP
> server's certificate)
> I think there should an extra pair of properties to specify the
> truststore/password for LDAP server, and use that to configure system
> properties {{javax.net.ssl.trustStore}}/{{javax.net.ssl.trustStorePassword}}
> I am a security layman so my words can be imprecise. But I hope this makes
> sense.
> Oracle's SSL LDAP documentation:
> http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ssl.html
> JSSE reference guide:
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15253) Should update maxQueueSize when refresh call queue
[
https://issues.apache.org/jira/browse/HADOOP-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-15253:
-
Target Version/s: 2.7.6
Fix Version/s: (was: 2.7.6)
> Should update maxQueueSize when refresh call queue
> --
>
> Key: HADOOP-15253
> URL: https://issues.apache.org/jira/browse/HADOOP-15253
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.8.2
>Reporter: Tao Jie
>Assignee: Tao Jie
>Priority: Minor
> Attachments: HADOOP-15253.001.patch, HADOOP-15253.002.patch,
> HADOOP-15253.003.patch
>
>
> When calling {{dfsadmin -refreshCallQueue}} to update CallQueue instance,
> {{maxQueueSize}} should also be updated.
> In case of changing CallQueue instance to FairCallQueue, the length of each
> queue in FairCallQueue would be 1/priorityLevels of original length of
> DefaultCallQueue. So it would be helpful for us to set the length of
> callqueue to a proper value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15253) Should update maxQueueSize when refresh call queue
[
https://issues.apache.org/jira/browse/HADOOP-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-15253:
-
Fix Version/s: 2.7.6
> Should update maxQueueSize when refresh call queue
> --
>
> Key: HADOOP-15253
> URL: https://issues.apache.org/jira/browse/HADOOP-15253
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.8.2
>Reporter: Tao Jie
>Assignee: Tao Jie
>Priority: Minor
> Fix For: 2.7.6
>
> Attachments: HADOOP-15253.001.patch, HADOOP-15253.002.patch,
> HADOOP-15253.003.patch
>
>
> When calling {{dfsadmin -refreshCallQueue}} to update CallQueue instance,
> {{maxQueueSize}} should also be updated.
> In case of changing CallQueue instance to FairCallQueue, the length of each
> queue in FairCallQueue would be 1/priorityLevels of original length of
> DefaultCallQueue. So it would be helpful for us to set the length of
> callqueue to a proper value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15253) Should update maxQueueSize when refresh call queue
[
https://issues.apache.org/jira/browse/HADOOP-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407111#comment-16407111
]
Konstantin Shvachko commented on HADOOP-15253:
--
+1 this looks good. Will commit in a bit.
What is the target version for this?
> Should update maxQueueSize when refresh call queue
> --
>
> Key: HADOOP-15253
> URL: https://issues.apache.org/jira/browse/HADOOP-15253
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.8.2
>Reporter: Tao Jie
>Assignee: Tao Jie
>Priority: Minor
> Attachments: HADOOP-15253.001.patch, HADOOP-15253.002.patch,
> HADOOP-15253.003.patch
>
>
> When calling {{dfsadmin -refreshCallQueue}} to update CallQueue instance,
> {{maxQueueSize}} should also be updated.
> In case of changing CallQueue instance to FairCallQueue, the length of each
> queue in FairCallQueue would be 1/priorityLevels of original length of
> DefaultCallQueue. So it would be helpful for us to set the length of
> callqueue to a proper value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15253) Should update maxQueueSize when refresh call queue
[
https://issues.apache.org/jira/browse/HADOOP-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405657#comment-16405657
]
Konstantin Shvachko commented on HADOOP-15253:
--
The change looks good to me. Minor things for the unit test:
# import of {{DFSConfigKeys}} is redundant.
# Would be better to incorporate the check of the queue size change into
{{testRefresh()}} instead of creating a new test case. That way the test will
not start up mini cluster one more time, will run faster.
> Should update maxQueueSize when refresh call queue
> --
>
> Key: HADOOP-15253
> URL: https://issues.apache.org/jira/browse/HADOOP-15253
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.8.2
>Reporter: Tao Jie
>Assignee: Tao Jie
>Priority: Minor
> Attachments: HADOOP-15253.001.patch, HADOOP-15253.002.patch
>
>
> When calling {{dfsadmin -refreshCallQueue}} to update CallQueue instance,
> {{maxQueueSize}} should also be updated.
> In case of changing CallQueue instance to FairCallQueue, the length of each
> queue in FairCallQueue would be 1/priorityLevels of original length of
> DefaultCallQueue. So it would be helpful for us to set the length of
> callqueue to a proper value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12862) LDAP Group Mapping over SSL can not specify trust store
[
https://issues.apache.org/jira/browse/HADOOP-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403075#comment-16403075
]
Konstantin Shvachko commented on HADOOP-12862:
--
Sounds like testing is a longer-term issue. BTW if I look into Hadoop
dependencies I see apacheds and ldapsdk. May they be useful for testing. I
wouldn't know.
The testing went well in our environment.
What do you think about removing {{.ssl.truststore.password}}? I really think
people should not use configs for passwords. Typically configs are checked in
in git repositories, so having passwords there is even worth than printing them
on a command line, which [as you
suggested|https://issues.apache.org/jira/browse/HADOOP-15315?focusedCommentId=16399166=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16399166]
is a bad practice.
> LDAP Group Mapping over SSL can not specify trust store
> ---
>
> Key: HADOOP-12862
> URL: https://issues.apache.org/jira/browse/HADOOP-12862
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HADOOP-12862.001.patch, HADOOP-12862.002.patch,
> HADOOP-12862.003.patch, HADOOP-12862.004.patch, HADOOP-12862.005.patch,
> HADOOP-12862.006.patch, HADOOP-12862.007.patch, HADOOP-12862.008.patch
>
>
> In a secure environment, SSL is used to encrypt LDAP request for group
> mapping resolution.
> We (+[~yoderme], +[~tgrayson]) have found that its implementation is strange.
> For information, Hadoop name node, as an LDAP client, talks to a LDAP server
> to resolve the group mapping of a user. In the case of LDAP over SSL, a
> typical scenario is to establish one-way authentication (the client verifies
> the server's certificate is real) by storing the server's certificate in the
> client's truststore.
> A rarer scenario is to establish two-way authentication: in addition to store
> truststore for the client to verify the server, the server also verifies the
> client's certificate is real, and the client stores its own certificate in
> its keystore.
> However, the current implementation for LDAP over SSL does not seem to be
> correct in that it only configures keystore but no truststore (so LDAP server
> can verify Hadoop's certificate, but Hadoop may not be able to verify LDAP
> server's certificate)
> I think there should an extra pair of properties to specify the
> truststore/password for LDAP server, and use that to configure system
> properties {{javax.net.ssl.trustStore}}/{{javax.net.ssl.trustStorePassword}}
> I am a security layman so my words can be imprecise. But I hope this makes
> sense.
> Oracle's SSL LDAP documentation:
> http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ssl.html
> JSSE reference guide:
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14687) AuthenticatedURL will reuse bad/expired session cookies
[
https://issues.apache.org/jira/browse/HADOOP-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401388#comment-16401388
]
Konstantin Shvachko commented on HADOOP-14687:
--
Just heads up here. The revert of HADOOP-13119 removed one test
{{testAuthenticationWithProxyUser()}} introduced here.
You might want to reinstate it somewhere.
> AuthenticatedURL will reuse bad/expired session cookies
> ---
>
> Key: HADOOP-14687
> URL: https://issues.apache.org/jira/browse/HADOOP-14687
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.6.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.2
>
> Attachments: HADOOP-14687.2.trunk.patch,
> HADOOP-14687.branch-2.8.patch, HADOOP-14687.trunk.patch
>
>
> AuthenticatedURL with kerberos was designed to perform spnego, then use a
> session cookie to avoid renegotiation overhead. Unfortunately the client
> will continue to use a cookie after it expires. Every request elicits a 401,
> connection closes (despite keepalive because 401 is an "error"), TGS is
> obtained, connection re-opened, re-requests with TGS, repeat cycle. This
> places a strain on the kdc and creates lots of time_wait sockets.
>
> The main problem is unbeknownst to the auth url, the JDK transparently does
> spnego. The server issues a new cookie but the auth url doesn't scrape the
> cookie from the response because it doesn't know the JDK re-authenticated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12862) LDAP Group Mapping over SSL can not specify trust store
[
https://issues.apache.org/jira/browse/HADOOP-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399562#comment-16399562
]
Konstantin Shvachko commented on HADOOP-12862:
--
Thanks for working on this [~jojochuang]. It is indeed the same issue as we hit
HADOOP-15315.
Two questions on the patch:
# Is there a way to unit test this? Seems the original implementation was wrong
since there was no test for it.
# I agree with [~drankye] we should not encourage people to store passwords in
core-site.xml. If we cannot remove {{ssl.keytstore.password}} due to
compatibility restrictions, we should at least break the patter and not
introduce {{ssl.truststore.password}} just the {{ssl.truststore.password.file}}.
Plan to do testing with your patch on our cluster.
> LDAP Group Mapping over SSL can not specify trust store
> ---
>
> Key: HADOOP-12862
> URL: https://issues.apache.org/jira/browse/HADOOP-12862
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HADOOP-12862.001.patch, HADOOP-12862.002.patch,
> HADOOP-12862.003.patch, HADOOP-12862.004.patch, HADOOP-12862.005.patch,
> HADOOP-12862.006.patch, HADOOP-12862.007.patch, HADOOP-12862.008.patch
>
>
> In a secure environment, SSL is used to encrypt LDAP request for group
> mapping resolution.
> We (+[~yoderme], +[~tgrayson]) have found that its implementation is strange.
> For information, Hadoop name node, as an LDAP client, talks to a LDAP server
> to resolve the group mapping of a user. In the case of LDAP over SSL, a
> typical scenario is to establish one-way authentication (the client verifies
> the server's certificate is real) by storing the server's certificate in the
> client's truststore.
> A rarer scenario is to establish two-way authentication: in addition to store
> truststore for the client to verify the server, the server also verifies the
> client's certificate is real, and the client stores its own certificate in
> its keystore.
> However, the current implementation for LDAP over SSL does not seem to be
> correct in that it only configures keystore but no truststore (so LDAP server
> can verify Hadoop's certificate, but Hadoop may not be able to verify LDAP
> server's certificate)
> I think there should an extra pair of properties to specify the
> truststore/password for LDAP server, and use that to configure system
> properties {{javax.net.ssl.trustStore}}/{{javax.net.ssl.trustStorePassword}}
> I am a security layman so my words can be imprecise. But I hope this makes
> sense.
> Oracle's SSL LDAP documentation:
> http://docs.oracle.com/javase/jndi/tutorial/ldap/security/ssl.html
> JSSE reference guide:
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13105) Support timeouts in LDAP queries in LdapGroupsMapping.
[
https://issues.apache.org/jira/browse/HADOOP-13105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-13105:
-
Fix Version/s: 2.7.6
I just ported this to branch-2.7. Updating Fix Versions.
This required HADOOP-12472 for tests to pass.
> Support timeouts in LDAP queries in LdapGroupsMapping.
> --
>
> Key: HADOOP-13105
> URL: https://issues.apache.org/jira/browse/HADOOP-13105
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
>Priority: Major
> Fix For: 2.8.0, 3.0.0-alpha1, 2.7.6
>
> Attachments: HADOOP-13105.000.patch, HADOOP-13105.001.patch,
> HADOOP-13105.002.patch, HADOOP-13105.003.patch, HADOOP-13105.004.patch
>
>
> {{LdapGroupsMapping}} currently does not set timeouts on the LDAP queries.
> This can create a risk of a very long/infinite wait on a connection.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12472) Make GenericTestUtils.assertExceptionContains robust
[
https://issues.apache.org/jira/browse/HADOOP-12472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-12472:
-
Fix Version/s: 2.7.6
I just committed this to branch-2.7. Updating Fix Versions.
> Make GenericTestUtils.assertExceptionContains robust
>
>
> Key: HADOOP-12472
> URL: https://issues.apache.org/jira/browse/HADOOP-12472
> Project: Hadoop Common
> Issue Type: Improvement
> Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha1, 2.7.6
>
> Attachments: HADOOP-12472-001.patch
>
>
> {{GenericTestUtils.assertExceptionContains}} calls
> {{Exception.getMessage()}}, followed by msg.contains().
> This will NPE for an exception with a null message, such as NPE.
> # it should call toString()
> # and do an assertNotNull on the result in case some subclass does something
> very bad
> # and for safety, check the asser
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12001) Limiting LDAP search conflicts with posixGroup addition
[ https://issues.apache.org/jira/browse/HADOOP-12001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HADOOP-12001: - Labels: (was: release-blocker) Fix Version/s: 2.7.6 I just committed this to branch-2.7. Updating Fix Versions. > Limiting LDAP search conflicts with posixGroup addition > --- > > Key: HADOOP-12001 > URL: https://issues.apache.org/jira/browse/HADOOP-12001 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.0, 2.8.0 >Reporter: Patrick White >Assignee: Patrick White >Priority: Blocker > Fix For: 2.8.0, 3.0.0-alpha1, 2.7.6 > > Attachments: HADOOP-12001.002.patch, HADOOP-12001.patch > > > In HADOOP-9477, posixGroup support was added > In HADOOP-10626, a limit on the returned attributes was added to speed up > queries. > Limiting the attributes can break the SEARCH_CONTROLS object in the context > of the isPosix block, since it only asks LDAP for the groupNameAttr -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12001) Limiting LDAP search conflicts with posixGroup addition
[ https://issues.apache.org/jira/browse/HADOOP-12001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HADOOP-12001: - Labels: release-blocker (was: ) Target Version/s: 2.7.6 (was: 2.8.0) > Limiting LDAP search conflicts with posixGroup addition > --- > > Key: HADOOP-12001 > URL: https://issues.apache.org/jira/browse/HADOOP-12001 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.0, 2.8.0 >Reporter: Patrick White >Assignee: Patrick White >Priority: Blocker > Labels: release-blocker > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HADOOP-12001.002.patch, HADOOP-12001.patch > > > In HADOOP-9477, posixGroup support was added > In HADOOP-10626, a limit on the returned attributes was added to speed up > queries. > Limiting the attributes can break the SEARCH_CONTROLS object in the context > of the isPosix block, since it only asks LDAP for the groupNameAttr -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12001) Limiting LDAP search conflicts with posixGroup addition
[ https://issues.apache.org/jira/browse/HADOOP-12001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388371#comment-16388371 ] Konstantin Shvachko commented on HADOOP-12001: -- Now that HADOOP-9477 is in 2.7.6, will need to backport this to 2.7 as well. Would have been easier, if it was properly linked. > Limiting LDAP search conflicts with posixGroup addition > --- > > Key: HADOOP-12001 > URL: https://issues.apache.org/jira/browse/HADOOP-12001 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.0, 2.8.0 >Reporter: Patrick White >Assignee: Patrick White >Priority: Blocker > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HADOOP-12001.002.patch, HADOOP-12001.patch > > > In HADOOP-9477, posixGroup support was added > In HADOOP-10626, a limit on the returned attributes was added to speed up > queries. > Limiting the attributes can break the SEARCH_CONTROLS object in the context > of the isPosix block, since it only asks LDAP for the groupNameAttr -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15205) maven release: missing source attachments for hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/HADOOP-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377894#comment-16377894 ] Konstantin Shvachko commented on HADOOP-15205: -- I checked on [Nexus|https://repository.apache.org/]. It looks like the releases were staged like that. See e.g. [https://repository.apache.org/content/repositories/releases/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.5/] * For 2.7.5 it's very strange, because some sub-projects like hadoop-common and hadoop-hdfs do have sources [https://repository.apache.org/content/repositories/releases/org/apache/hadoop/hadoop-common/2.7.5/] [https://repository.apache.org/content/repositories/releases/org/apache/hadoop/hadoop-hdfs/2.7.5/] while others don't. * For 2.8.3, 2.9.0, and 3.0.0 sources are consistently not there. * The 3.0.1 RC0, which is currently staged, seems to have all sources. Didn't check all sub-projects though https://repository.apache.org/content/repositories/orgapachehadoop-1078/org/apache/hadoop/hadoop-mapreduce-client-common/3.0.1/ I will try to debug this when I release 2.7.6. May be we need to update the [HowToRelease|https://wiki.apache.org/hadoop/HowToRelease] runbook. > maven release: missing source attachments for hadoop-mapreduce-client-core > -- > > Key: HADOOP-15205 > URL: https://issues.apache.org/jira/browse/HADOOP-15205 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.5, 3.0.0 >Reporter: Zoltan Haindrich >Priority: Major > > I wanted to use the source attachment; however it looks like since 2.7.5 that > artifact is not present at maven central ; it looks like the last release > which had source attachments / javadocs was 2.7.4 > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.5/ > this seems to be not limited to mapreduce; as the same change is present for > yarn-common as well > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.5/ > and also hadoop-common > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.5/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.0.0/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12568) Update core-default.xml to describe posixGroups support
[ https://issues.apache.org/jira/browse/HADOOP-12568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HADOOP-12568: - Fix Version/s: 2.7.6 Just pushed this into branch-2.7. Updating fix version. > Update core-default.xml to describe posixGroups support > --- > > Key: HADOOP-12568 > URL: https://issues.apache.org/jira/browse/HADOOP-12568 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Minor > Labels: group, mappings, supportability > Fix For: 2.8.0, 3.0.0-alpha1, 2.7.6 > > Attachments: HADOOP-12568.001.patch, HADOOP-12568.002.patch > > > After HADOOP-9477, LdapGroupsMapping supports posixGroups mapping service. > However, core-default.xml was not updated to detail how to configure in order > to enable this feature. This JIRA is filed to describe how to enable > posixGroups for users. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9477) Add posixGroups support for LDAP groups mapping service
[ https://issues.apache.org/jira/browse/HADOOP-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HADOOP-9477: Fix Version/s: 2.7.6 Just pushed this into branch-2.7. Updating fix version. > Add posixGroups support for LDAP groups mapping service > --- > > Key: HADOOP-9477 > URL: https://issues.apache.org/jira/browse/HADOOP-9477 > Project: Hadoop Common > Issue Type: New Feature >Affects Versions: 2.0.4-alpha >Reporter: Kai Zheng >Assignee: Dapeng Sun >Priority: Major > Fix For: 2.8.0, 3.0.0-alpha1, 2.7.6 > > Attachments: HADOOP-9477.003.patch, HADOOP-9477.004.patch, > HADOOP-9477.005.patch, HADOOP-9477.006.patch, HADOOP-9477.007.patch, > HADOOP-9477.008.patch, HADOOP-9477.009.patch, HADOOP-9477.patch, > HADOOP-9477.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > It would be nice to support posixGroups for LdapGroupsMapping service. Below > is from current description for the provider: > hadoop.security.group.mapping.ldap.search.filter.group: > An additional filter to use when searching for LDAP groups. This should be > changed when resolving groups against a non-Active Directory installation. > posixGroups are currently not a supported group class. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13508) FsPermission string constructor does not recognize sticky bit
[
https://issues.apache.org/jira/browse/HADOOP-13508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-13508:
-
Target Version/s: 3.0.0-alpha1, 2.9.0 (was: 2.9.0, 3.0.0-alpha1)
Fix Version/s: 2.7.6
I just committed this to branch-2.7. Thanks.
> FsPermission string constructor does not recognize sticky bit
> -
>
> Key: HADOOP-13508
> URL: https://issues.apache.org/jira/browse/HADOOP-13508
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Atul Sikaria
>Assignee: Atul Sikaria
>Priority: Major
> Fix For: 2.9.0, 3.0.0-alpha2, 2.8.2, 2.7.6
>
> Attachments: HADOOP-13508-1.patch, HADOOP-13508-2.patch,
> HADOOP-13508.003.patch, HADOOP-13508.004.patch, HADOOP-13508.005.patch,
> HADOOP-13508.006.patch, HADOOP-13508.branch-2.patch
>
>
> FsPermissions's string constructor breaks on valid permission strings, like
> "1777".
> This is because FsPermission class naïvely uses UmaskParser to do it’s
> parsing of permissions: (from source code):
> public FsPermission(String mode) {
> this((new UmaskParser(mode)).getUMask());
> }
> The mode string UMask accepts is subtly different (esp wrt sticky bit), so
> parsing Umask is not the same as parsing FsPermission.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13375) o.a.h.security.TestGroupsCaching.testBackgroundRefreshCounters seems flaky
[
https://issues.apache.org/jira/browse/HADOOP-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-13375:
-
Fix Version/s: 2.7.6
Just committed this to branch-2.7. Thanks.
> o.a.h.security.TestGroupsCaching.testBackgroundRefreshCounters seems flaky
> --
>
> Key: HADOOP-13375
> URL: https://issues.apache.org/jira/browse/HADOOP-13375
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, test
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Weiwei Yang
>Priority: Major
> Fix For: 2.8.0, 3.0.0-alpha2, 2.7.6
>
> Attachments: HADOOP-13375.001.patch, HADOOP-13375.002.patch,
> HADOOP-13375.003.patch, HADOOP-13375.004.patch, HADOOP-13375.005.patch,
> HADOOP-13375.006.patch, HADOOP-13375.007.patch
>
>
> h5. Error Message
> bq. expected:<1> but was:<0>
> h5. Stacktrace
> {quote}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.security.TestGroupsCaching.testBackgroundRefreshCounters(TestGroupsCaching.java:638)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13263) Reload cached groups in background after expiry
[ https://issues.apache.org/jira/browse/HADOOP-13263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HADOOP-13263: - Fix Version/s: 2.7.6 Just committed this to branch-2.7. Thanks. > Reload cached groups in background after expiry > --- > > Key: HADOOP-13263 > URL: https://issues.apache.org/jira/browse/HADOOP-13263 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Fix For: 2.8.0, 3.0.0-alpha1, 2.7.6 > > Attachments: HADOOP-13263.001.patch, HADOOP-13263.002.patch, > HADOOP-13263.003.patch, HADOOP-13263.004.patch, HADOOP-13263.005.patch, > HADOOP-13263.006.patch, HADOOP-13263.007.patch > > > In HADOOP-11238 the Guava cache was introduced to allow refreshes on the > Namenode group cache to run in the background, avoiding many slow group > lookups. Even with this change, I have seen quite a few clusters with issues > due to slow group lookups. The problem is most prevalent in HA clusters, > where a slow group lookup on the hdfs user can fail to return for over 45 > seconds causing the Failover Controller to kill it. > The way the current Guava cache implementation works is approximately: > 1) On initial load, the first thread to request groups for a given user > blocks until it returns. Any subsequent threads requesting that user block > until that first thread populates the cache. > 2) When the key expires, the first thread to hit the cache after expiry > blocks. While it is blocked, other threads will return the old value. > I feel it is this blocking thread that still gives the Namenode issues on > slow group lookups. If the call from the FC is the one that blocks and > lookups are slow, if can cause the NN to be killed. > Guava has the ability to refresh expired keys completely in the background, > where the first thread that hits an expired key schedules a background cache > reload, but still returns the old value. Then the cache is eventually > updated. This patch introduces this background reload feature. There are two > new parameters: > 1) hadoop.security.groups.cache.background.reload - default false to keep the > current behaviour. Set to true to enable a small thread pool and background > refresh for expired keys > 2) hadoop.security.groups.cache.background.reload.threads - only relevant if > the above is set to true. Controls how many threads are in the background > refresh pool. Default is 1, which is likely to be enough. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12751) While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple
[ https://issues.apache.org/jira/browse/HADOOP-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HADOOP-12751: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.7.6 Status: Resolved (was: Patch Available) Just committed this to branch-2.7. Thanks. > While using kerberos Hadoop incorrectly assumes names with '@' to be > non-simple > --- > > Key: HADOOP-12751 > URL: https://issues.apache.org/jira/browse/HADOOP-12751 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.2 > Environment: kerberos >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Labels: kerberos > Fix For: 2.7.6, 3.0.0-alpha1, 2.8.0 > > Attachments: 0001-HADOOP-12751-leave-user-validation-to-os.patch, > 0001-Remove-check-for-user-name-characters-and.patch, > 0002-HADOOP-12751-leave-user-validation-to-os.patch, > 0003-HADOOP-12751-leave-user-validation-to-os.patch, > 0004-HADOOP-12751-leave-user-validation-to-os.patch, > 0005-HADOOP-12751-leave-user-validation-to-os.patch, > 0006-HADOOP-12751-leave-user-validation-to-os.patch, > 0007-HADOOP-12751-leave-user-validation-to-os.patch, > 0007-HADOOP-12751-leave-user-validation-to-os.patch, > 0008-HADOOP-12751-leave-user-validation-to-os.patch, > 0008-HADOOP-12751-leave-user-validation-to-os.patch, HADOOP-12751-009.patch, > HADOOP-12751-branch-2.7.009.patch > > > In the scenario of a trust between two directories, eg. FreeIPA (ipa.local) > and Active Directory (ad.local) users can be made available on the OS level > by something like sssd. The trusted users will be of the form 'user@ad.local' > while other users are will not contain the domain. Executing 'id -Gn > user@ad.local' will successfully return the groups the user belongs to if > configured correctly. > However, it is assumed by Hadoop that users of the format with '@' cannot be > correct. This code is in KerberosName.java and seems to be a validator if the > 'auth_to_local' rules are applied correctly. > In my opinion this should be removed or changed to a different kind of check > or maybe logged as a warning while still proceeding, as the current behavior > limits integration possibilities with other standard tools. > Workaround are difficult to apply (by having a rewrite by system tools to for > example user_ad_local) due to down stream consequences. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12751) While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple
[ https://issues.apache.org/jira/browse/HADOOP-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331429#comment-16331429 ] Konstantin Shvachko edited comment on HADOOP-12751 at 1/18/18 11:42 PM: Thanks for the review [~brahmareddy]. Yes Yetus is trying to download Oracle Java 8, which probably requires a login. It should be OpenJDK7 for branch-2.7. Something that should have been fixed by HADOOP-14474? was (Author: shv): Thanks for the review [~brahmareddy]. Yes Yetus is trying to download Oracle Java 8, which probably requires a login. It should be OpenJDK2 for branch-2.7. Something that should have been fixed by HADOOP-14474? > While using kerberos Hadoop incorrectly assumes names with '@' to be > non-simple > --- > > Key: HADOOP-12751 > URL: https://issues.apache.org/jira/browse/HADOOP-12751 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.2 > Environment: kerberos >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Labels: kerberos > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: 0001-HADOOP-12751-leave-user-validation-to-os.patch, > 0001-Remove-check-for-user-name-characters-and.patch, > 0002-HADOOP-12751-leave-user-validation-to-os.patch, > 0003-HADOOP-12751-leave-user-validation-to-os.patch, > 0004-HADOOP-12751-leave-user-validation-to-os.patch, > 0005-HADOOP-12751-leave-user-validation-to-os.patch, > 0006-HADOOP-12751-leave-user-validation-to-os.patch, > 0007-HADOOP-12751-leave-user-validation-to-os.patch, > 0007-HADOOP-12751-leave-user-validation-to-os.patch, > 0008-HADOOP-12751-leave-user-validation-to-os.patch, > 0008-HADOOP-12751-leave-user-validation-to-os.patch, HADOOP-12751-009.patch, > HADOOP-12751-branch-2.7.009.patch > > > In the scenario of a trust between two directories, eg. FreeIPA (ipa.local) > and Active Directory (ad.local) users can be made available on the OS level > by something like sssd. The trusted users will be of the form 'user@ad.local' > while other users are will not contain the domain. Executing 'id -Gn > user@ad.local' will successfully return the groups the user belongs to if > configured correctly. > However, it is assumed by Hadoop that users of the format with '@' cannot be > correct. This code is in KerberosName.java and seems to be a validator if the > 'auth_to_local' rules are applied correctly. > In my opinion this should be removed or changed to a different kind of check > or maybe logged as a warning while still proceeding, as the current behavior > limits integration possibilities with other standard tools. > Workaround are difficult to apply (by having a rewrite by system tools to for > example user_ad_local) due to down stream consequences. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12751) While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple
[ https://issues.apache.org/jira/browse/HADOOP-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331429#comment-16331429 ] Konstantin Shvachko commented on HADOOP-12751: -- Thanks for the review [~brahmareddy]. Yes Yetus is trying to download Oracle Java 8, which probably requires a login. It should be OpenJDK2 for branch-2.7. Something that should have been fixed by HADOOP-14474? > While using kerberos Hadoop incorrectly assumes names with '@' to be > non-simple > --- > > Key: HADOOP-12751 > URL: https://issues.apache.org/jira/browse/HADOOP-12751 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.2 > Environment: kerberos >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Critical > Labels: kerberos > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: 0001-HADOOP-12751-leave-user-validation-to-os.patch, > 0001-Remove-check-for-user-name-characters-and.patch, > 0002-HADOOP-12751-leave-user-validation-to-os.patch, > 0003-HADOOP-12751-leave-user-validation-to-os.patch, > 0004-HADOOP-12751-leave-user-validation-to-os.patch, > 0005-HADOOP-12751-leave-user-validation-to-os.patch, > 0006-HADOOP-12751-leave-user-validation-to-os.patch, > 0007-HADOOP-12751-leave-user-validation-to-os.patch, > 0007-HADOOP-12751-leave-user-validation-to-os.patch, > 0008-HADOOP-12751-leave-user-validation-to-os.patch, > 0008-HADOOP-12751-leave-user-validation-to-os.patch, HADOOP-12751-009.patch, > HADOOP-12751-branch-2.7.009.patch > > > In the scenario of a trust between two directories, eg. FreeIPA (ipa.local) > and Active Directory (ad.local) users can be made available on the OS level > by something like sssd. The trusted users will be of the form 'user@ad.local' > while other users are will not contain the domain. Executing 'id -Gn > user@ad.local' will successfully return the groups the user belongs to if > configured correctly. > However, it is assumed by Hadoop that users of the format with '@' cannot be > correct. This code is in KerberosName.java and seems to be a validator if the > 'auth_to_local' rules are applied correctly. > In my opinion this should be removed or changed to a different kind of check > or maybe logged as a warning while still proceeding, as the current behavior > limits integration possibilities with other standard tools. > Workaround are difficult to apply (by having a rewrite by system tools to for > example user_ad_local) due to down stream consequences. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12751) While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple
[
https://issues.apache.org/jira/browse/HADOOP-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329726#comment-16329726
]
Konstantin Shvachko edited comment on HADOOP-12751 at 1/18/18 12:09 AM:
Attached a patch for branch-2.7. Cherry picked all changes from 2.8 commit
except {{KDiag}}, because diagnostic class is not in 2.7.
LMK if that works.
was (Author: shv):
Attached a patch for branch-2.7. Cherry picked all changes from 2.8 commit
except {{KDiag}}, because diagnostic class is not in 2.7.
LMK of that works.
> While using kerberos Hadoop incorrectly assumes names with '@' to be
> non-simple
> ---
>
> Key: HADOOP-12751
> URL: https://issues.apache.org/jira/browse/HADOOP-12751
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.7.2
> Environment: kerberos
>Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Critical
> Labels: kerberos
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: 0001-HADOOP-12751-leave-user-validation-to-os.patch,
> 0001-Remove-check-for-user-name-characters-and.patch,
> 0002-HADOOP-12751-leave-user-validation-to-os.patch,
> 0003-HADOOP-12751-leave-user-validation-to-os.patch,
> 0004-HADOOP-12751-leave-user-validation-to-os.patch,
> 0005-HADOOP-12751-leave-user-validation-to-os.patch,
> 0006-HADOOP-12751-leave-user-validation-to-os.patch,
> 0007-HADOOP-12751-leave-user-validation-to-os.patch,
> 0007-HADOOP-12751-leave-user-validation-to-os.patch,
> 0008-HADOOP-12751-leave-user-validation-to-os.patch,
> 0008-HADOOP-12751-leave-user-validation-to-os.patch, HADOOP-12751-009.patch,
> HADOOP-12751-branch-2.7.009.patch
>
>
> In the scenario of a trust between two directories, eg. FreeIPA (ipa.local)
> and Active Directory (ad.local) users can be made available on the OS level
> by something like sssd. The trusted users will be of the form 'user@ad.local'
> while other users are will not contain the domain. Executing 'id -Gn
> user@ad.local' will successfully return the groups the user belongs to if
> configured correctly.
> However, it is assumed by Hadoop that users of the format with '@' cannot be
> correct. This code is in KerberosName.java and seems to be a validator if the
> 'auth_to_local' rules are applied correctly.
> In my opinion this should be removed or changed to a different kind of check
> or maybe logged as a warning while still proceeding, as the current behavior
> limits integration possibilities with other standard tools.
> Workaround are difficult to apply (by having a rewrite by system tools to for
> example user_ad_local) due to down stream consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12751) While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple
[
https://issues.apache.org/jira/browse/HADOOP-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-12751:
-
Status: Patch Available (was: Reopened)
Attached a patch for branch-2.7. Cherry picked all changes from 2.8 commit
except {{KDiag}}, because diagnostic class is not in 2.7.
LMK of that works.
> While using kerberos Hadoop incorrectly assumes names with '@' to be
> non-simple
> ---
>
> Key: HADOOP-12751
> URL: https://issues.apache.org/jira/browse/HADOOP-12751
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.7.2
> Environment: kerberos
>Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Critical
> Labels: kerberos
> Fix For: 3.0.0-alpha1, 2.8.0
>
> Attachments: 0001-HADOOP-12751-leave-user-validation-to-os.patch,
> 0001-Remove-check-for-user-name-characters-and.patch,
> 0002-HADOOP-12751-leave-user-validation-to-os.patch,
> 0003-HADOOP-12751-leave-user-validation-to-os.patch,
> 0004-HADOOP-12751-leave-user-validation-to-os.patch,
> 0005-HADOOP-12751-leave-user-validation-to-os.patch,
> 0006-HADOOP-12751-leave-user-validation-to-os.patch,
> 0007-HADOOP-12751-leave-user-validation-to-os.patch,
> 0007-HADOOP-12751-leave-user-validation-to-os.patch,
> 0008-HADOOP-12751-leave-user-validation-to-os.patch,
> 0008-HADOOP-12751-leave-user-validation-to-os.patch, HADOOP-12751-009.patch,
> HADOOP-12751-branch-2.7.009.patch
>
>
> In the scenario of a trust between two directories, eg. FreeIPA (ipa.local)
> and Active Directory (ad.local) users can be made available on the OS level
> by something like sssd. The trusted users will be of the form 'user@ad.local'
> while other users are will not contain the domain. Executing 'id -Gn
> user@ad.local' will successfully return the groups the user belongs to if
> configured correctly.
> However, it is assumed by Hadoop that users of the format with '@' cannot be
> correct. This code is in KerberosName.java and seems to be a validator if the
> 'auth_to_local' rules are applied correctly.
> In my opinion this should be removed or changed to a different kind of check
> or maybe logged as a warning while still proceeding, as the current behavior
> limits integration possibilities with other standard tools.
> Workaround are difficult to apply (by having a rewrite by system tools to for
> example user_ad_local) due to down stream consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
