[jira] Updated: (HADOOP-6943) The GroupMappingServiceProvider interface should be public
[ https://issues.apache.org/jira/browse/HADOOP-6943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-6943: --- Attachment: hadoop-6943.txt.1 Thanks for the comments, Tom. I've updated the patch to address them. The GroupMappingServiceProvider interface should be public -- Key: HADOOP-6943 URL: https://issues.apache.org/jira/browse/HADOOP-6943 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 0.22.0 Reporter: Aaron T. Myers Attachments: hadoop-6943.txt, hadoop-6943.txt.1 The GroupMappingServiceProvider interface is presently package-protected. It seems likely that many organizations will be implementing their own versions of this to suit their particular setup. It would be nice if this interface were made public, and annotated with @InterfaceAudience.Private and @InterfaceStability.Evolving. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6879) Provide SSH based (Jsch) remote execution API for system tests
[
https://issues.apache.org/jira/browse/HADOOP-6879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik updated HADOOP-6879:
---
Attachment: HADOOP-6879.y20.patch
HADOOP-6879.patch
Base implementing method should use {{portNumber}} variable instead of a
constant.
Provide SSH based (Jsch) remote execution API for system tests
--
Key: HADOOP-6879
URL: https://issues.apache.org/jira/browse/HADOOP-6879
Project: Hadoop Common
Issue Type: Improvement
Components: build, test
Affects Versions: 0.22.0
Reporter: Iyappan Srinivasan
Assignee: Iyappan Srinivasan
Attachments: 6879-ydist-security-patch.txt, HADOOP-6879.patch,
HADOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.patch,
HADOOP-6879.patch, HADOOP-6879.y20.patch, HADOOP-6879.y20.patch,
HADOOP-6879.y20.patch
http://mvnrepository.com/
com.jcraft ยป jsch
0.1.42 version needs to be included in the build. This is needed to
facilitate implementation of some system (Herriot) testcases .
Please include this in ivy.
jsch is originally located in http://www.jcraft.com/jsch/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12909368#action_12909368 ] Aaron T. Myers commented on HADOOP-6857: bq. We already have fs -count path which counts bytes including replications. Is it good enough? fs -count path is indeed sufficient. Feel free to close this ticket. FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Fix For: 0.22.0 Attachments: show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-6951) Distinct minicluster services (e.g. NN and JT) overwrite each other's service policies
[ https://issues.apache.org/jira/browse/HADOOP-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers reassigned HADOOP-6951: -- Assignee: Aaron T. Myers Distinct minicluster services (e.g. NN and JT) overwrite each other's service policies -- Key: HADOOP-6951 URL: https://issues.apache.org/jira/browse/HADOOP-6951 Project: Hadoop Common Issue Type: Bug Components: security Reporter: Aaron T. Myers Assignee: Aaron T. Myers Because the protocol - ACL mapping in ServiceAuthorizationManager is static, services which are run in the same JVM have the potential to clobber the other's service authorization ACLs whenever ServiceAuthorizationManager.refresh() is called. This causes authorization failures if one tries to launch a 2NN connected to a minicluster with hadoop.security.authorization enabled. Seems like each service should have its own instance of a ServiceAuthorizationManager, instead of using static methods. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-6943) The GroupMappingServiceProvider interface should be public
[ https://issues.apache.org/jira/browse/HADOOP-6943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers reassigned HADOOP-6943: -- Assignee: Aaron T. Myers The GroupMappingServiceProvider interface should be public -- Key: HADOOP-6943 URL: https://issues.apache.org/jira/browse/HADOOP-6943 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 0.22.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: hadoop-6943.txt, hadoop-6943.txt.1 The GroupMappingServiceProvider interface is presently package-protected. It seems likely that many organizations will be implementing their own versions of this to suit their particular setup. It would be nice if this interface were made public, and annotated with @InterfaceAudience.Private and @InterfaceStability.Evolving. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6857) FsShell should report raw disk usage including replication factor
[
https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12909472#action_12909472
]
Eli Collins commented on HADOOP-6857:
-
Why do we have separate dus and count commands? They seem to duplicate
each other. Since the CLI needs to be backwards compatible not suggesting we
remove one, but perhaps we should update the hdfs_shell docs to make it clear
that they display the same info if there are not significant differences.
{code}
~ $ hadoop fs -dus /user/eli
hdfs://haus01.sf.cloudera.com:10020/user/eli86183666860
{code}
{code}
~ $ hadoop fs -count /user/eli
7 5186183666860
hdfs://haus01.sf.cloudera.com:10020/user/eli
~ $
{code}
FsShell should report raw disk usage including replication factor
-
Key: HADOOP-6857
URL: https://issues.apache.org/jira/browse/HADOOP-6857
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Reporter: Alex Kozlov
Fix For: 0.22.0
Attachments: show-space-consumed.txt
Currently FsShell report HDFS usage with hadoop fs -dus path command.
Since replication level is per file level, it would be nice to add raw disk
usage including the replication factor (maybe hadoop fs -dus -raw path?).
This will allow to assess resource usage more accurately. -- Alex K
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12909478#action_12909478 ] Tsz Wo (Nicholas), SZE commented on HADOOP-6857: fs -count was introduced for counting name objects. The disk space column was added later on. FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Fix For: 0.22.0 Attachments: show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6857) FsShell should report raw disk usage including replication factor
[
https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12909502#action_12909502
]
Koji Noguchi commented on HADOOP-6857:
--
A little confused. I thought fs -count shows hdfs usage same as fs -du at
the thrid column.
{noformat}
[knoguchi ~]$ hadoop dfs -dus /user/knoguchi
hdfs://abc-nn1.com/user/knoguchi 2603203340273
[knoguchi ~]$ hadoop dfs -count /user/knoguchi
158020624 2603203340273 hdfs://abc-nn1.com/user/knoguchi
[knoguchi ~]$
{noformat}
If quota is enabled on that dir and -q is passed, it would show the remaining
raw space available.
{noformat}
[knoguchi ~]$ hadoop dfs -count -q /user/knoguchi
5 27796 13194139533312 5384528402193 1580
20624 2603203340273 hdfs://abc-nn1.com/user/knoguchi
[knoguchi ~]$
{noformat}
You can get the raw space usage then. (quota - raw\_remaining).
However *this is only if you have quota enabled on that particular dir*.
FsShell should report raw disk usage including replication factor
-
Key: HADOOP-6857
URL: https://issues.apache.org/jira/browse/HADOOP-6857
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Reporter: Alex Kozlov
Fix For: 0.22.0
Attachments: show-space-consumed.txt
Currently FsShell report HDFS usage with hadoop fs -dus path command.
Since replication level is per file level, it would be nice to add raw disk
usage including the replication factor (maybe hadoop fs -dus -raw path?).
This will allow to assess resource usage more accurately. -- Alex K
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi reopened HADOOP-6857: -- I think this number(raw usage) would be helpful. Not sure whether this should be in -du or -count and by default or as an option. FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Fix For: 0.22.0 Attachments: show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6951) Distinct minicluster services (e.g. NN and JT) overwrite each other's service policies
[ https://issues.apache.org/jira/browse/HADOOP-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-6951: --- Attachment: hadoop-6951.txt.0 Change ServiceAuthorizationManager to have a non-static map, and change Server to include an instance of ServiceAuthorizationManager. Distinct minicluster services (e.g. NN and JT) overwrite each other's service policies -- Key: HADOOP-6951 URL: https://issues.apache.org/jira/browse/HADOOP-6951 Project: Hadoop Common Issue Type: Bug Components: security Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: hadoop-6951.txt.0 Because the protocol - ACL mapping in ServiceAuthorizationManager is static, services which are run in the same JVM have the potential to clobber the other's service authorization ACLs whenever ServiceAuthorizationManager.refresh() is called. This causes authorization failures if one tries to launch a 2NN connected to a minicluster with hadoop.security.authorization enabled. Seems like each service should have its own instance of a ServiceAuthorizationManager, instead of using static methods. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6951) Distinct minicluster services (e.g. NN and JT) overwrite each other's service policies
[ https://issues.apache.org/jira/browse/HADOOP-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-6951: --- Status: Patch Available (was: Open) Fix Version/s: 0.22.0 Distinct minicluster services (e.g. NN and JT) overwrite each other's service policies -- Key: HADOOP-6951 URL: https://issues.apache.org/jira/browse/HADOOP-6951 Project: Hadoop Common Issue Type: Bug Components: security Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.22.0 Attachments: hadoop-6951.txt.0 Because the protocol - ACL mapping in ServiceAuthorizationManager is static, services which are run in the same JVM have the potential to clobber the other's service authorization ACLs whenever ServiceAuthorizationManager.refresh() is called. This causes authorization failures if one tries to launch a 2NN connected to a minicluster with hadoop.security.authorization enabled. Seems like each service should have its own instance of a ServiceAuthorizationManager, instead of using static methods. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6952) Support sending priority RPC
Support sending priority RPC Key: HADOOP-6952 URL: https://issues.apache.org/jira/browse/HADOOP-6952 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: dhruba borthakur Assignee: dhruba borthakur There are certain class of RPCs that need priority delivery. This applies especially to heartbeat RPCs that distributed systems (like HDFS) uses. Ability to deliver heartbeat RPCs earlier than other data-movement RPCs can improve the scalability of large scale distributed systems. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
