[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13954992#comment-13954992 ] Akira AJISAKA commented on HADOOP-8989: --- The patch mostly looks good. I built with the patch and verified some expressions worked. This feature is also very useful to search files/dirs in a fsimage with new OfflineImageViewer, and therefore I want this patch will be merged in. Minor nit: There's some trailing whitespaces. hadoop dfs -find feature Key: HADOOP-8989 URL: https://issues.apache.org/jira/browse/HADOOP-8989 Project: Hadoop Common Issue Type: New Feature Reporter: Marco Nicosia Assignee: Jonathan Allen Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch Both sysadmins and users make frequent use of the unix 'find' command, but Hadoop has no correlate. Without this, users are writing scripts which make heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs -lsr is somewhat taxing on the NameNode, and a really slow experience on the client side. Possibly an in-NameNode find operation would be only a bit more taxing on the NameNode, but significantly faster from the client's point of view? The minimum set of options I can think of which would make a Hadoop find command generally useful is (in priority order): * -type (file or directory, for now) * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) * -print0 (for piping to xargs -0) * -depth * -owner/-group (and -nouser/-nogroup) * -name (allowing for shell pattern, or even regex?) * -perm * -size One possible special case, but could possibly be really cool if it ran from within the NameNode: * -delete The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow. Lower priority, some people do use operators, mostly to execute -or searches such as: * find / \(-nouser -or -nogroup\) Finally, I thought I'd include a link to the [Posix spec for find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9907) Webapp http://hostname:port/metrics link is not working
[ https://issues.apache.org/jira/browse/HADOOP-9907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958718#comment-13958718 ] Akira AJISAKA commented on HADOOP-9907: --- I verified /metrics link is not working and /jmx?qry=Hadoop:* works with Hadoop 1.2.1. Should I create a patch for branch-1 also? Webapp http://hostname:port/metrics link is not working - Key: HADOOP-9907 URL: https://issues.apache.org/jira/browse/HADOOP-9907 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.3.0 Reporter: Jian He Assignee: Akira AJISAKA Priority: Critical Attachments: HADOOP-9907.patch This link is not working which just shows a blank page. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10460) Please update on-line documentation for hadoop 2.3
[ https://issues.apache.org/jira/browse/HADOOP-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959523#comment-13959523 ] Akira AJISAKA commented on HADOOP-10460: SingleNodeSetup.html is deprecated and not linked from the left-side menu page. You can use SingleCluster.html for single node setup. SingleCluster.html is also deprecated, however, it will be updated in HADOOP-10139 (2.4.0). You can download 2.4.0-rc0 (http://people.apache.org/~acmurthy/hadoop-2.4.0-rc0/) or go to my documentation build (http://aajisaka.github.io/hadoop-project/hadoop-project-dist/hadoop-common/SingleCluster.html) to get the new document. By the way, I suggest to remove SingleNodeSetup.html, which makes users confusing. I'll create a patch shortly. Please update on-line documentation for hadoop 2.3 -- Key: HADOOP-10460 URL: https://issues.apache.org/jira/browse/HADOOP-10460 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 2.3.0 Environment: any Reporter: Darek Documentation on page: http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/SingleNodeSetup.html contains steps like: $ cp conf/*.xml input but after checked out repository conf does not exists (I quess it was moved to etc/hadoop) Few lines below in section Execution there are steps: $ bin/hadoop namenode -format - OK $ bin/start-all.sh - this file has been removed -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HADOOP-10460) Please update on-line documentation for hadoop 2.3
[ https://issues.apache.org/jira/browse/HADOOP-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA reassigned HADOOP-10460: -- Assignee: Akira AJISAKA Please update on-line documentation for hadoop 2.3 -- Key: HADOOP-10460 URL: https://issues.apache.org/jira/browse/HADOOP-10460 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 2.3.0 Environment: any Reporter: Darek Assignee: Akira AJISAKA Documentation on page: http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/SingleNodeSetup.html contains steps like: $ cp conf/*.xml input but after checked out repository conf does not exists (I quess it was moved to etc/hadoop) Few lines below in section Execution there are steps: $ bin/hadoop namenode -format - OK $ bin/start-all.sh - this file has been removed -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10460) Please update on-line documentation for hadoop 2.3
[ https://issues.apache.org/jira/browse/HADOOP-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10460: --- Labels: newbie (was: ) Target Version/s: 2.4.1 Status: Patch Available (was: Open) Please update on-line documentation for hadoop 2.3 -- Key: HADOOP-10460 URL: https://issues.apache.org/jira/browse/HADOOP-10460 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 2.3.0 Environment: any Reporter: Darek Assignee: Akira AJISAKA Labels: newbie Attachments: HADOOP-10460.patch Documentation on page: http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/SingleNodeSetup.html contains steps like: $ cp conf/*.xml input but after checked out repository conf does not exists (I quess it was moved to etc/hadoop) Few lines below in section Execution there are steps: $ bin/hadoop namenode -format - OK $ bin/start-all.sh - this file has been removed -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10460) Please update on-line documentation for hadoop 2.3
[ https://issues.apache.org/jira/browse/HADOOP-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10460: --- Attachment: HADOOP-10460.patch Attaching a patch. Please update on-line documentation for hadoop 2.3 -- Key: HADOOP-10460 URL: https://issues.apache.org/jira/browse/HADOOP-10460 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 2.3.0 Environment: any Reporter: Darek Assignee: Akira AJISAKA Labels: newbie Attachments: HADOOP-10460.patch Documentation on page: http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/SingleNodeSetup.html contains steps like: $ cp conf/*.xml input but after checked out repository conf does not exists (I quess it was moved to etc/hadoop) Few lines below in section Execution there are steps: $ bin/hadoop namenode -format - OK $ bin/start-all.sh - this file has been removed -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Moved] (HADOOP-10462) NameNodeResourceChecker prints 'null' mount point to the log
[
https://issues.apache.org/jira/browse/HADOOP-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA moved HDFS-6073 to HADOOP-10462:
--
Component/s: (was: namenode)
Target Version/s: (was: 2.4.0)
Affects Version/s: (was: 2.3.0)
2.3.0
Key: HADOOP-10462 (was: HDFS-6073)
Project: Hadoop Common (was: Hadoop HDFS)
NameNodeResourceChecker prints 'null' mount point to the log
Key: HADOOP-10462
URL: https://issues.apache.org/jira/browse/HADOOP-10462
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.3.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Attachments: HDFS-6073.2.patch, HDFS-6073.patch
If the available space on the volume used for saving fsimage is less than
100MB (default), NameNodeResourceChecker prints the log as follows:
{code}
Space available on volume 'null' is 92274688, which is below the configured
reserved amount 104857600
{code}
It should print an appropriate mount point instead of null.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10462) DF#getFilesystem is not parsing the command output
[
https://issues.apache.org/jira/browse/HADOOP-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10462:
---
Summary: DF#getFilesystem is not parsing the command output (was:
NameNodeResourceChecker prints 'null' mount point to the log)
DF#getFilesystem is not parsing the command output
--
Key: HADOOP-10462
URL: https://issues.apache.org/jira/browse/HADOOP-10462
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.3.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Attachments: HDFS-6073.2.patch, HDFS-6073.patch
If the available space on the volume used for saving fsimage is less than
100MB (default), NameNodeResourceChecker prints the log as follows:
{code}
Space available on volume 'null' is 92274688, which is below the configured
reserved amount 104857600
{code}
It should print an appropriate mount point instead of null.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10462) DF#getFilesystem is not parsing the command output
[
https://issues.apache.org/jira/browse/HADOOP-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10462:
---
Description:
{{DF#getFileSystem}} returns null if {{DF#getMount}} is not called before. This
is because {{DF#getFileSystem}} is not parsing the df command output.
This causes {{NameNodeResourceChecker}} to print the log as follows if the
available space on the volume used for saving fsimage is less than the
specified value (100MB by default).
{code}
Space available on volume 'null' is 92274688, which is below the configured
reserved amount 104857600
{code}
was:
If the available space on the volume used for saving fsimage is less than 100MB
(default), NameNodeResourceChecker prints the log as follows:
{code}
Space available on volume 'null' is 92274688, which is below the configured
reserved amount 104857600
{code}
It should print an appropriate mount point instead of null.
DF#getFilesystem is not parsing the command output
--
Key: HADOOP-10462
URL: https://issues.apache.org/jira/browse/HADOOP-10462
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.3.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Attachments: HDFS-6073.2.patch, HDFS-6073.patch
{{DF#getFileSystem}} returns null if {{DF#getMount}} is not called before.
This is because {{DF#getFileSystem}} is not parsing the df command output.
This causes {{NameNodeResourceChecker}} to print the log as follows if the
available space on the volume used for saving fsimage is less than the
specified value (100MB by default).
{code}
Space available on volume 'null' is 92274688, which is below the configured
reserved amount 104857600
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10462) DF#getFilesystem is not parsing the command output
[
https://issues.apache.org/jira/browse/HADOOP-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10462:
---
Attachment: HADOOP-10462.3.patch
Attaching a patch that separates {{TestDFVariations#TestMountAndFileSystem}}
and removes unused import from {{DF}} and {{TestDFVariations}}.
DF#getFilesystem is not parsing the command output
--
Key: HADOOP-10462
URL: https://issues.apache.org/jira/browse/HADOOP-10462
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.3.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Attachments: HADOOP-10462.3.patch, HDFS-6073.2.patch, HDFS-6073.patch
{{DF#getFileSystem}} returns null if {{DF#getMount}} is not called before.
This is because {{DF#getFileSystem}} is not parsing the df command output.
This causes {{NameNodeResourceChecker}} to print the log as follows if the
available space on the volume used for saving fsimage is less than the
specified value (100MB by default).
{code}
Space available on volume 'null' is 92274688, which is below the configured
reserved amount 104857600
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10104) update jackson
[ https://issues.apache.org/jira/browse/HADOOP-10104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10104: --- Attachment: HADOOP-10104.4.patch Thanks Steve for retrying this. Updated the patch to fix the new deprecation warnings. update jackson -- Key: HADOOP-10104 URL: https://issues.apache.org/jira/browse/HADOOP-10104 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 2.2.0, 2.3.0, 2.4.0 Reporter: Steve Loughran Assignee: Akira AJISAKA Priority: Minor Attachments: HADOOP-10104-003.patch, HADOOP-10104.2.patch, HADOOP-10104.4.patch, HADOOP-10104.patch Jackson is now at 1.9.13, [apparently|http://mvnrepository.com/artifact/org.codehaus.jackson/jackson-core-asl], hadoop 2.2 at 1.8.8. jackson isn't used that much in the code so risk from an update *should* be low -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[
https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969272#comment-13969272
]
Akira AJISAKA commented on HADOOP-8989:
---
Thanks [~jonallen] for updating the patch.
bq. How about splitting this to small patches?
+1 (non-binding) for splitting. If you don't have a time to do this, I'm
willing to help.
{code}
/**
* Construct a Print {@Expression} with an operational ASCII NULL
* suffix.
*/
private Print(boolean appendNull) {
{code}
{{@Expression}} should be {{@link Expression}}.
{code}
public int apply(int current, int shift, int value) {
return current | (value shift);
};
{code}
In {{Perm.java}} some semicolons can be removed.
hadoop dfs -find feature
Key: HADOOP-8989
URL: https://issues.apache.org/jira/browse/HADOOP-8989
Project: Hadoop Common
Issue Type: New Feature
Reporter: Marco Nicosia
Assignee: Jonathan Allen
Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
HADOOP-8989.patch
Both sysadmins and users make frequent use of the unix 'find' command, but
Hadoop has no correlate. Without this, users are writing scripts which make
heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs
-lsr is somewhat taxing on the NameNode, and a really slow experience on the
client side. Possibly an in-NameNode find operation would be only a bit more
taxing on the NameNode, but significantly faster from the client's point of
view?
The minimum set of options I can think of which would make a Hadoop find
command generally useful is (in priority order):
* -type (file or directory, for now)
* -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments)
* -print0 (for piping to xargs -0)
* -depth
* -owner/-group (and -nouser/-nogroup)
* -name (allowing for shell pattern, or even regex?)
* -perm
* -size
One possible special case, but could possibly be really cool if it ran from
within the NameNode:
* -delete
The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow.
Lower priority, some people do use operators, mostly to execute -or searches
such as:
* find / \(-nouser -or -nogroup\)
Finally, I thought I'd include a link to the [Posix spec for
find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html]
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970295#comment-13970295 ] Akira AJISAKA commented on HADOOP-8989: --- I suggest to create extra issues. You can create sub-tasks and attach a small patch on each issue. hadoop dfs -find feature Key: HADOOP-8989 URL: https://issues.apache.org/jira/browse/HADOOP-8989 Project: Hadoop Common Issue Type: New Feature Reporter: Marco Nicosia Assignee: Jonathan Allen Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch Both sysadmins and users make frequent use of the unix 'find' command, but Hadoop has no correlate. Without this, users are writing scripts which make heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs -lsr is somewhat taxing on the NameNode, and a really slow experience on the client side. Possibly an in-NameNode find operation would be only a bit more taxing on the NameNode, but significantly faster from the client's point of view? The minimum set of options I can think of which would make a Hadoop find command generally useful is (in priority order): * -type (file or directory, for now) * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) * -print0 (for piping to xargs -0) * -depth * -owner/-group (and -nouser/-nogroup) * -name (allowing for shell pattern, or even regex?) * -perm * -size One possible special case, but could possibly be really cool if it ran from within the NameNode: * -delete The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow. Lower priority, some people do use operators, mostly to execute -or searches such as: * find / \(-nouser -or -nogroup\) Finally, I thought I'd include a link to the [Posix spec for find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-8808) Update FsShell documentation to mention deprecation of some of the commands, and mention alternatives
[ https://issues.apache.org/jira/browse/HADOOP-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-8808: -- Target Version/s: 2.5.0 (was: 2.4.0) Update FsShell documentation to mention deprecation of some of the commands, and mention alternatives - Key: HADOOP-8808 URL: https://issues.apache.org/jira/browse/HADOOP-8808 Project: Hadoop Common Issue Type: Bug Components: documentation, fs Affects Versions: 2.2.0 Reporter: Hemanth Yamijala Assignee: Akira AJISAKA Attachments: HADOOP-8808.2.patch, HADOOP-8808.patch In HADOOP-7286, we deprecated the following 3 commands dus, lsr and rmr, in favour of du -s, ls -r and rm -r respectively. The FsShell documentation should be updated to mention these, so that users can start switching. Also, there are places where we refer to the deprecated commands as alternatives. This can be changed as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9907) Webapp http://hostname:port/metrics link is not working
[ https://issues.apache.org/jira/browse/HADOOP-9907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-9907: -- Target Version/s: 2.5.0 (was: 2.4.0) Webapp http://hostname:port/metrics link is not working - Key: HADOOP-9907 URL: https://issues.apache.org/jira/browse/HADOOP-9907 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.3.0 Reporter: Jian He Assignee: Akira AJISAKA Priority: Critical Attachments: HADOOP-9907.patch This link is not working which just shows a blank page. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10392) Use FileSystem#makeQualified(Path) instead of Path#makeQualified(FileSystem)
[ https://issues.apache.org/jira/browse/HADOOP-10392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10392: --- Target Version/s: 2.5.0 (was: 2.4.0) Use FileSystem#makeQualified(Path) instead of Path#makeQualified(FileSystem) Key: HADOOP-10392 URL: https://issues.apache.org/jira/browse/HADOOP-10392 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.3.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Labels: newbie Attachments: HADOOP-10392.2.patch, HADOOP-10392.3.patch, HADOOP-10392.4.patch, HADOOP-10392.patch There're some methods calling Path.makeQualified(FileSystem), which causes javac warning. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13973912#comment-13973912 ] Akira AJISAKA commented on HADOOP-8989: --- I think you can add a depends upon link to an issue to clarify the order. ex.) If the patch 2 at issue 2 depends on the patch 1 at issue 1, it's better to add depends upon issue 1 link to issue 2. hadoop dfs -find feature Key: HADOOP-8989 URL: https://issues.apache.org/jira/browse/HADOOP-8989 Project: Hadoop Common Issue Type: New Feature Reporter: Marco Nicosia Assignee: Jonathan Allen Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch Both sysadmins and users make frequent use of the unix 'find' command, but Hadoop has no correlate. Without this, users are writing scripts which make heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs -lsr is somewhat taxing on the NameNode, and a really slow experience on the client side. Possibly an in-NameNode find operation would be only a bit more taxing on the NameNode, but significantly faster from the client's point of view? The minimum set of options I can think of which would make a Hadoop find command generally useful is (in priority order): * -type (file or directory, for now) * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) * -print0 (for piping to xargs -0) * -depth * -owner/-group (and -nouser/-nogroup) * -name (allowing for shell pattern, or even regex?) * -perm * -size One possible special case, but could possibly be really cool if it ran from within the NameNode: * -delete The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow. Lower priority, some people do use operators, mostly to execute -or searches such as: * find / \(-nouser -or -nogroup\) Finally, I thought I'd include a link to the [Posix spec for find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9919) Rewrite hadoop-metrics2.properties
[
https://issues.apache.org/jira/browse/HADOOP-9919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-9919:
--
Target Version/s: 3.0.0, 2.5.0 (was: 3.0.0)
Rewrite hadoop-metrics2.properties
--
Key: HADOOP-9919
URL: https://issues.apache.org/jira/browse/HADOOP-9919
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 2.1.0-beta
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Attachments: HADOOP-9919.2.patch, HADOOP-9919.3.patch,
HADOOP-9919.4.patch, HADOOP-9919.patch
The config for JobTracker and TaskTracker (comment outed) still exists in
hadoop-metrics2.properties as follows:
{code}
#jobtracker.sink.file_jvm.context=jvm
#jobtracker.sink.file_jvm.filename=jobtracker-jvm-metrics.out
#jobtracker.sink.file_mapred.context=mapred
#jobtracker.sink.file_mapred.filename=jobtracker-mapred-metrics.out
#tasktracker.sink.file.filename=tasktracker-metrics.out
{code}
These lines should be removed and a config for NodeManager should be added
instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10525) Remove DRFA.MaxBackupIndex config from log4j.properties
[
https://issues.apache.org/jira/browse/HADOOP-10525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10525:
---
Summary: Remove DRFA.MaxBackupIndex config from log4j.properties (was:
Remove MaxBackupIndex config from DailyRollingFileAppender)
Remove DRFA.MaxBackupIndex config from log4j.properties
---
Key: HADOOP-10525
URL: https://issues.apache.org/jira/browse/HADOOP-10525
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.4.0
Reporter: Akira AJISAKA
Priority: Minor
Labels: newbie
From [hadoop-user mailing
list|http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3C534FACD3.8040907%40corp.badoo.com%3E].
{code}
# 30-day backup
# log4j.appender.DRFA.MaxBackupIndex=30
{code}
In {{log4j.properties}}, the above lines should be removed because
DailyRollingFileAppender(DRFA) doesn't support MaxBackupIndex config.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10525) Remove MaxBackupIndex config from DailyRollingFileAppender
Akira AJISAKA created HADOOP-10525:
--
Summary: Remove MaxBackupIndex config from DailyRollingFileAppender
Key: HADOOP-10525
URL: https://issues.apache.org/jira/browse/HADOOP-10525
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.4.0
Reporter: Akira AJISAKA
Priority: Minor
From [hadoop-user mailing
list|http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3C534FACD3.8040907%40corp.badoo.com%3E].
{code}
# 30-day backup
# log4j.appender.DRFA.MaxBackupIndex=30
{code}
In {{log4j.properties}}, the above lines should be removed because
DailyRollingFileAppender(DRFA) doesn't support MaxBackupIndex config.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10525) Remove DRFA.MaxBackupIndex config from log4j.properties
[
https://issues.apache.org/jira/browse/HADOOP-10525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10525:
---
Attachment: HADOOP-10525.patch
Attaching a patch.
Remove DRFA.MaxBackupIndex config from log4j.properties
---
Key: HADOOP-10525
URL: https://issues.apache.org/jira/browse/HADOOP-10525
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.4.0
Reporter: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: HADOOP-10525.patch
From [hadoop-user mailing
list|http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3C534FACD3.8040907%40corp.badoo.com%3E].
{code}
# 30-day backup
# log4j.appender.DRFA.MaxBackupIndex=30
{code}
In {{log4j.properties}}, the above lines should be removed because
DailyRollingFileAppender(DRFA) doesn't support MaxBackupIndex config.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10525) Remove DRFA.MaxBackupIndex config from log4j.properties
[
https://issues.apache.org/jira/browse/HADOOP-10525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10525:
---
Assignee: Akira AJISAKA
Target Version/s: 2.5.0
Status: Patch Available (was: Open)
Remove DRFA.MaxBackupIndex config from log4j.properties
---
Key: HADOOP-10525
URL: https://issues.apache.org/jira/browse/HADOOP-10525
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.4.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: HADOOP-10525.patch
From [hadoop-user mailing
list|http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3C534FACD3.8040907%40corp.badoo.com%3E].
{code}
# 30-day backup
# log4j.appender.DRFA.MaxBackupIndex=30
{code}
In {{log4j.properties}}, the above lines should be removed because
DailyRollingFileAppender(DRFA) doesn't support MaxBackupIndex config.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9919) Rewrite hadoop-metrics2.properties
[
https://issues.apache.org/jira/browse/HADOOP-9919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13975417#comment-13975417
]
Akira AJISAKA commented on HADOOP-9919:
---
The patch is to update only the comment, so new tests are not needed.
Rewrite hadoop-metrics2.properties
--
Key: HADOOP-9919
URL: https://issues.apache.org/jira/browse/HADOOP-9919
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 2.1.0-beta
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Attachments: HADOOP-9919.2.patch, HADOOP-9919.3.patch,
HADOOP-9919.4.patch, HADOOP-9919.patch
The config for JobTracker and TaskTracker (comment outed) still exists in
hadoop-metrics2.properties as follows:
{code}
#jobtracker.sink.file_jvm.context=jvm
#jobtracker.sink.file_jvm.filename=jobtracker-jvm-metrics.out
#jobtracker.sink.file_mapred.context=mapred
#jobtracker.sink.file_mapred.filename=jobtracker-mapred-metrics.out
#tasktracker.sink.file.filename=tasktracker-metrics.out
{code}
These lines should be removed and a config for NodeManager should be added
instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10525) Remove DRFA.MaxBackupIndex config from log4j.properties
[
https://issues.apache.org/jira/browse/HADOOP-10525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13975426#comment-13975426
]
Akira AJISAKA commented on HADOOP-10525:
The patch is just to remove the comments, so new tests are not needed.
Remove DRFA.MaxBackupIndex config from log4j.properties
---
Key: HADOOP-10525
URL: https://issues.apache.org/jira/browse/HADOOP-10525
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.4.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: HADOOP-10525.patch
From [hadoop-user mailing
list|http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3C534FACD3.8040907%40corp.badoo.com%3E].
{code}
# 30-day backup
# log4j.appender.DRFA.MaxBackupIndex=30
{code}
In {{log4j.properties}}, the above lines should be removed because
DailyRollingFileAppender(DRFA) doesn't support MaxBackupIndex config.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9238) FsShell -put from stdin auto-creates paths
[ https://issues.apache.org/jira/browse/HADOOP-9238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977857#comment-13977857 ] Akira AJISAKA commented on HADOOP-9238: --- Hi [~sarutak], what's going on this jira? I'd like to take it over. FsShell -put from stdin auto-creates paths -- Key: HADOOP-9238 URL: https://issues.apache.org/jira/browse/HADOOP-9238 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Attachments: HADOOP-9238.patch FsShell put is no longer supposed to auto-create paths. There's an inconsistency where a put from stdin will still auto-create paths. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[
https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13986396#comment-13986396
]
Akira AJISAKA commented on HADOOP-8989:
---
Thanks [~jonallen] for splitting the patch!
{quote}
I think you get the results as full paths
{code}
hdfs://127.0.0.1:59894/texttest
hdfs://127.0.0.1:59894/texttest/file..bz2
hdfs://127.0.0.1:59894/texttest/file.deflate
hdfs://127.0.0.1:59894/texttest/file.gz
{code}
This should be relative paths I think from where you are finding, no?
{quote}
Would you please reflect the above [~umamaheswararao]'s comment?
A new comment:
* The tests for -iname option is needed.
Two minor nits:
* Typo in TestAnd.java
{code}
// test both expressions failining
{code}
* There're some lines over 80 characters in TestFind.java
hadoop dfs -find feature
Key: HADOOP-8989
URL: https://issues.apache.org/jira/browse/HADOOP-8989
Project: Hadoop Common
Issue Type: New Feature
Reporter: Marco Nicosia
Assignee: Jonathan Allen
Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch
Both sysadmins and users make frequent use of the unix 'find' command, but
Hadoop has no correlate. Without this, users are writing scripts which make
heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs
-lsr is somewhat taxing on the NameNode, and a really slow experience on the
client side. Possibly an in-NameNode find operation would be only a bit more
taxing on the NameNode, but significantly faster from the client's point of
view?
The minimum set of options I can think of which would make a Hadoop find
command generally useful is (in priority order):
* -type (file or directory, for now)
* -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments)
* -print0 (for piping to xargs -0)
* -depth
* -owner/-group (and -nouser/-nogroup)
* -name (allowing for shell pattern, or even regex?)
* -perm
* -size
One possible special case, but could possibly be really cool if it ran from
within the NameNode:
* -delete
The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow.
Lower priority, some people do use operators, mostly to execute -or searches
such as:
* find / \(-nouser -or -nogroup\)
Finally, I thought I'd include a link to the [Posix spec for
find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html]
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-9705:
--
Target Version/s: 2.5.0 (was: 2.4.0)
FsShell cp -p does not preserve directory attibutes
---
Key: HADOOP-9705
URL: https://issues.apache.org/jira/browse/HADOOP-9705
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Stephen Chu
Assignee: Akira AJISAKA
Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
HADOOP-9705.patch
HADOOP-9338 added the -p flag to preserve file attributes when copying.
However, cp -p does not preserve directory attributes. It'd be useful to add
this functionality.
For example, the following shows that the modified time is not preserved
{code}
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
/user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
Found 1 items
drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
/user/schu/testDir1 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
Found 2 items
drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$
{code}
The preserve logic is in CommandWithDestination#copyFileToTarget, which is
only called with files.
{code}
protected void processPath(PathData src, PathData dst) throws IOException {
if (src.stat.isSymlink()) {
// TODO: remove when FileContext is supported, this needs to either
// copy the symlink or deref the symlink
throw new PathOperationException(src.toString());
} else if (src.stat.isFile()) {
copyFileToTarget(src, dst);
} else if (src.stat.isDirectory() !isRecursive()) {
throw new PathIsDirectoryException(src.toString());
}
}
{code}
{code}
/**
* Copies the source file to the target.
* @param src item to copy
* @param target where to copy the item
* @throws IOException if copy fails
*/
protected void copyFileToTarget(PathData src, PathData target) throws
IOException {
src.fs.setVerifyChecksum(verifyChecksum);
if (src != null) {
throw new PathExistsException(hi);
}
InputStream in = null;
try {
in = src.fs.open(src.path);
copyStreamToTarget(in, target);
if(preserve) {
target.fs.setTimes(
target.path,
src.stat.getModificationTime(),
src.stat.getAccessTime());
target.fs.setOwner(
target.path,
src.stat.getOwner(),
src.stat.getGroup());
target.fs.setPermission(
target.path,
src.stat.getPermission());
System.out.println(Preserving);
if (src.fs.equals(target.fs)) {
System.out.println(Same filesystems);
src.fs.preserveAttributes(src.path, target.path);
}
throw new IOException(hi);
}
} finally {
IOUtils.closeStream(in);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
Akira AJISAKA created HADOOP-10557: -- Summary: FsShell -cp -p does not preserve extended ACLs Key: HADOOP-10557 URL: https://issues.apache.org/jira/browse/HADOOP-10557 Project: Hadoop Common Issue Type: Bug Reporter: Akira AJISAKA This issue tracks enhancing FsShell cp to * preserve extended ACLs by -p option or * add a new command-line option for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[
https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13986436#comment-13986436
]
Akira AJISAKA commented on HADOOP-10557:
I perfer to preserve extended ACLs by {{-p}} option rather than creating a new
option because {{cp -p}} preserves extended ACLs in unix.
FsShell -cp -p does not preserve extended ACLs
--
Key: HADOOP-10557
URL: https://issues.apache.org/jira/browse/HADOOP-10557
Project: Hadoop Common
Issue Type: Bug
Reporter: Akira AJISAKA
This issue tracks enhancing FsShell cp to
* preserve extended ACLs by -p option
or
* add a new command-line option for preserving extended ACLs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA reassigned HADOOP-10557: -- Assignee: Akira AJISAKA FsShell -cp -p does not preserve extended ACLs -- Key: HADOOP-10557 URL: https://issues.apache.org/jira/browse/HADOOP-10557 Project: Hadoop Common Issue Type: Bug Reporter: Akira AJISAKA Assignee: Akira AJISAKA This issue tracks enhancing FsShell cp to * preserve extended ACLs by -p option or * add a new command-line option for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-9705:
--
Attachment: HADOOP-9705.4.patch
Moved the duplicated code into private method.
FsShell cp -p does not preserve directory attibutes
---
Key: HADOOP-9705
URL: https://issues.apache.org/jira/browse/HADOOP-9705
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Stephen Chu
Assignee: Akira AJISAKA
Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
HADOOP-9705.4.patch, HADOOP-9705.patch
HADOOP-9338 added the -p flag to preserve file attributes when copying.
However, cp -p does not preserve directory attributes. It'd be useful to add
this functionality.
For example, the following shows that the modified time is not preserved
{code}
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
/user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
Found 1 items
drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
/user/schu/testDir1 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
Found 2 items
drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$
{code}
The preserve logic is in CommandWithDestination#copyFileToTarget, which is
only called with files.
{code}
protected void processPath(PathData src, PathData dst) throws IOException {
if (src.stat.isSymlink()) {
// TODO: remove when FileContext is supported, this needs to either
// copy the symlink or deref the symlink
throw new PathOperationException(src.toString());
} else if (src.stat.isFile()) {
copyFileToTarget(src, dst);
} else if (src.stat.isDirectory() !isRecursive()) {
throw new PathIsDirectoryException(src.toString());
}
}
{code}
{code}
/**
* Copies the source file to the target.
* @param src item to copy
* @param target where to copy the item
* @throws IOException if copy fails
*/
protected void copyFileToTarget(PathData src, PathData target) throws
IOException {
src.fs.setVerifyChecksum(verifyChecksum);
if (src != null) {
throw new PathExistsException(hi);
}
InputStream in = null;
try {
in = src.fs.open(src.path);
copyStreamToTarget(in, target);
if(preserve) {
target.fs.setTimes(
target.path,
src.stat.getModificationTime(),
src.stat.getAccessTime());
target.fs.setOwner(
target.path,
src.stat.getOwner(),
src.stat.getGroup());
target.fs.setPermission(
target.path,
src.stat.getPermission());
System.out.println(Preserving);
if (src.fs.equals(target.fs)) {
System.out.println(Same filesystems);
src.fs.preserveAttributes(src.path, target.path);
}
throw new IOException(hi);
}
} finally {
IOUtils.closeStream(in);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-9705:
--
Attachment: HADOOP-9705.4.patch
FsShell cp -p does not preserve directory attibutes
---
Key: HADOOP-9705
URL: https://issues.apache.org/jira/browse/HADOOP-9705
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Stephen Chu
Assignee: Akira AJISAKA
Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
HADOOP-9705.4.patch, HADOOP-9705.patch
HADOOP-9338 added the -p flag to preserve file attributes when copying.
However, cp -p does not preserve directory attributes. It'd be useful to add
this functionality.
For example, the following shows that the modified time is not preserved
{code}
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
/user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
Found 1 items
drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
/user/schu/testDir1 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
Found 2 items
drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$
{code}
The preserve logic is in CommandWithDestination#copyFileToTarget, which is
only called with files.
{code}
protected void processPath(PathData src, PathData dst) throws IOException {
if (src.stat.isSymlink()) {
// TODO: remove when FileContext is supported, this needs to either
// copy the symlink or deref the symlink
throw new PathOperationException(src.toString());
} else if (src.stat.isFile()) {
copyFileToTarget(src, dst);
} else if (src.stat.isDirectory() !isRecursive()) {
throw new PathIsDirectoryException(src.toString());
}
}
{code}
{code}
/**
* Copies the source file to the target.
* @param src item to copy
* @param target where to copy the item
* @throws IOException if copy fails
*/
protected void copyFileToTarget(PathData src, PathData target) throws
IOException {
src.fs.setVerifyChecksum(verifyChecksum);
if (src != null) {
throw new PathExistsException(hi);
}
InputStream in = null;
try {
in = src.fs.open(src.path);
copyStreamToTarget(in, target);
if(preserve) {
target.fs.setTimes(
target.path,
src.stat.getModificationTime(),
src.stat.getAccessTime());
target.fs.setOwner(
target.path,
src.stat.getOwner(),
src.stat.getGroup());
target.fs.setPermission(
target.path,
src.stat.getPermission());
System.out.println(Preserving);
if (src.fs.equals(target.fs)) {
System.out.println(Same filesystems);
src.fs.preserveAttributes(src.path, target.path);
}
throw new IOException(hi);
}
} finally {
IOUtils.closeStream(in);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-9705:
--
Attachment: (was: HADOOP-9705.4.patch)
FsShell cp -p does not preserve directory attibutes
---
Key: HADOOP-9705
URL: https://issues.apache.org/jira/browse/HADOOP-9705
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Stephen Chu
Assignee: Akira AJISAKA
Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
HADOOP-9705.4.patch, HADOOP-9705.patch
HADOOP-9338 added the -p flag to preserve file attributes when copying.
However, cp -p does not preserve directory attributes. It'd be useful to add
this functionality.
For example, the following shows that the modified time is not preserved
{code}
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
/user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
Found 1 items
drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
/user/schu/testDir1 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
Found 2 items
drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$
{code}
The preserve logic is in CommandWithDestination#copyFileToTarget, which is
only called with files.
{code}
protected void processPath(PathData src, PathData dst) throws IOException {
if (src.stat.isSymlink()) {
// TODO: remove when FileContext is supported, this needs to either
// copy the symlink or deref the symlink
throw new PathOperationException(src.toString());
} else if (src.stat.isFile()) {
copyFileToTarget(src, dst);
} else if (src.stat.isDirectory() !isRecursive()) {
throw new PathIsDirectoryException(src.toString());
}
}
{code}
{code}
/**
* Copies the source file to the target.
* @param src item to copy
* @param target where to copy the item
* @throws IOException if copy fails
*/
protected void copyFileToTarget(PathData src, PathData target) throws
IOException {
src.fs.setVerifyChecksum(verifyChecksum);
if (src != null) {
throw new PathExistsException(hi);
}
InputStream in = null;
try {
in = src.fs.open(src.path);
copyStreamToTarget(in, target);
if(preserve) {
target.fs.setTimes(
target.path,
src.stat.getModificationTime(),
src.stat.getAccessTime());
target.fs.setOwner(
target.path,
src.stat.getOwner(),
src.stat.getGroup());
target.fs.setPermission(
target.path,
src.stat.getPermission());
System.out.println(Preserving);
if (src.fs.equals(target.fs)) {
System.out.println(Same filesystems);
src.fs.preserveAttributes(src.path, target.path);
}
throw new IOException(hi);
}
} finally {
IOUtils.closeStream(in);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992670#comment-13992670
]
Akira AJISAKA commented on HADOOP-9705:
---
The failure looks unrelated to the patch.
FsShell cp -p does not preserve directory attibutes
---
Key: HADOOP-9705
URL: https://issues.apache.org/jira/browse/HADOOP-9705
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Stephen Chu
Assignee: Akira AJISAKA
Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
HADOOP-9705.4.patch, HADOOP-9705.patch
HADOOP-9338 added the -p flag to preserve file attributes when copying.
However, cp -p does not preserve directory attributes. It'd be useful to add
this functionality.
For example, the following shows that the modified time is not preserved
{code}
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
/user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
Found 1 items
drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
/user/schu/testDir1 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
Found 2 items
drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$
{code}
The preserve logic is in CommandWithDestination#copyFileToTarget, which is
only called with files.
{code}
protected void processPath(PathData src, PathData dst) throws IOException {
if (src.stat.isSymlink()) {
// TODO: remove when FileContext is supported, this needs to either
// copy the symlink or deref the symlink
throw new PathOperationException(src.toString());
} else if (src.stat.isFile()) {
copyFileToTarget(src, dst);
} else if (src.stat.isDirectory() !isRecursive()) {
throw new PathIsDirectoryException(src.toString());
}
}
{code}
{code}
/**
* Copies the source file to the target.
* @param src item to copy
* @param target where to copy the item
* @throws IOException if copy fails
*/
protected void copyFileToTarget(PathData src, PathData target) throws
IOException {
src.fs.setVerifyChecksum(verifyChecksum);
if (src != null) {
throw new PathExistsException(hi);
}
InputStream in = null;
try {
in = src.fs.open(src.path);
copyStreamToTarget(in, target);
if(preserve) {
target.fs.setTimes(
target.path,
src.stat.getModificationTime(),
src.stat.getAccessTime());
target.fs.setOwner(
target.path,
src.stat.getOwner(),
src.stat.getGroup());
target.fs.setPermission(
target.path,
src.stat.getPermission());
System.out.println(Preserving);
if (src.fs.equals(target.fs)) {
System.out.println(Same filesystems);
src.fs.preserveAttributes(src.path, target.path);
}
throw new IOException(hi);
}
} finally {
IOUtils.closeStream(in);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HADOOP-10602) Documentation has broken Go Back hyperlinks.
[ https://issues.apache.org/jira/browse/HADOOP-10602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA reassigned HADOOP-10602: -- Assignee: Akira AJISAKA Documentation has broken Go Back hyperlinks. -- Key: HADOOP-10602 URL: https://issues.apache.org/jira/browse/HADOOP-10602 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 3.0.0, 2.4.0 Reporter: Chris Nauroth Assignee: Akira AJISAKA Priority: Trivial Labels: newbie Multiple pages of our documentation have Go Back links that are broken, because they point to an incorrect relative path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10602) Documentation has broken Go Back hyperlinks.
[ https://issues.apache.org/jira/browse/HADOOP-10602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10602: --- Status: Patch Available (was: Open) Documentation has broken Go Back hyperlinks. -- Key: HADOOP-10602 URL: https://issues.apache.org/jira/browse/HADOOP-10602 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 2.4.0, 3.0.0 Reporter: Chris Nauroth Assignee: Akira AJISAKA Priority: Trivial Labels: newbie Attachments: HADOOP-10602.patch Multiple pages of our documentation have Go Back links that are broken, because they point to an incorrect relative path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10602) Documentation has broken Go Back hyperlinks.
[ https://issues.apache.org/jira/browse/HADOOP-10602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10602: --- Attachment: HADOOP-10602.patch Attaching a patch. Documentation has broken Go Back hyperlinks. -- Key: HADOOP-10602 URL: https://issues.apache.org/jira/browse/HADOOP-10602 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 3.0.0, 2.4.0 Reporter: Chris Nauroth Assignee: Akira AJISAKA Priority: Trivial Labels: newbie Attachments: HADOOP-10602.patch Multiple pages of our documentation have Go Back links that are broken, because they point to an incorrect relative path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10392) Use FileSystem#makeQualified(Path) instead of Path#makeQualified(FileSystem)
[ https://issues.apache.org/jira/browse/HADOOP-10392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10392: --- Attachment: HADOOP-10392.4.patch Use FileSystem#makeQualified(Path) instead of Path#makeQualified(FileSystem) Key: HADOOP-10392 URL: https://issues.apache.org/jira/browse/HADOOP-10392 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.3.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Labels: newbie Attachments: HADOOP-10392.2.patch, HADOOP-10392.3.patch, HADOOP-10392.4.patch, HADOOP-10392.4.patch, HADOOP-10392.patch There're some methods calling Path.makeQualified(FileSystem), which causes javac warning. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10609) .gitignore should ignore .orig and .rej files
[ https://issues.apache.org/jira/browse/HADOOP-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998698#comment-13998698 ] Akira AJISAKA commented on HADOOP-10609: +1 (non-binding), make sense to me. .gitignore should ignore .orig and .rej files - Key: HADOOP-10609 URL: https://issues.apache.org/jira/browse/HADOOP-10609 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: hadoop-10609.patch .gitignore file should ignore .orig and .rej files -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10602) Documentation has broken Go Back hyperlinks.
[ https://issues.apache.org/jira/browse/HADOOP-10602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001380#comment-14001380 ] Akira AJISAKA commented on HADOOP-10602: Thanks [~cnauroth] for the comment. The approach of the v1 patch was to delete the only dead links. bq. we have all the direct links in the left nav, and the user always has the browser back button too. Now I agree with you. I'll update the patch to remove all the Go Back links. By the way, removing all the Go Back links makes hadoop-yarn/hadoop-yarn-site/index.html not linked from anywhere. I think we can remove the document because hadoop-yarn/hadoop-yarn-site/YARN.html is a superset of it. Documentation has broken Go Back hyperlinks. -- Key: HADOOP-10602 URL: https://issues.apache.org/jira/browse/HADOOP-10602 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 3.0.0, 2.4.0 Reporter: Chris Nauroth Assignee: Akira AJISAKA Priority: Trivial Labels: newbie Attachments: HADOOP-10602.patch Multiple pages of our documentation have Go Back links that are broken, because they point to an incorrect relative path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10602) Documentation has broken Go Back hyperlinks.
[ https://issues.apache.org/jira/browse/HADOOP-10602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10602: --- Attachment: HADOOP-10602.2.patch Documentation has broken Go Back hyperlinks. -- Key: HADOOP-10602 URL: https://issues.apache.org/jira/browse/HADOOP-10602 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 3.0.0, 2.4.0 Reporter: Chris Nauroth Assignee: Akira AJISAKA Priority: Trivial Labels: newbie Attachments: HADOOP-10602.2.patch, HADOOP-10602.patch Multiple pages of our documentation have Go Back links that are broken, because they point to an incorrect relative path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10602) Documentation has broken Go Back hyperlinks.
[ https://issues.apache.org/jira/browse/HADOOP-10602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001384#comment-14001384 ] Akira AJISAKA commented on HADOOP-10602: Attached a patch to remove all the Go Back links. Documentation has broken Go Back hyperlinks. -- Key: HADOOP-10602 URL: https://issues.apache.org/jira/browse/HADOOP-10602 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 3.0.0, 2.4.0 Reporter: Chris Nauroth Assignee: Akira AJISAKA Priority: Trivial Labels: newbie Attachments: HADOOP-10602.2.patch, HADOOP-10602.patch Multiple pages of our documentation have Go Back links that are broken, because they point to an incorrect relative path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10618) Remove SingleNodeSetup.apt.vm
Akira AJISAKA created HADOOP-10618: -- Summary: Remove SingleNodeSetup.apt.vm Key: HADOOP-10618 URL: https://issues.apache.org/jira/browse/HADOOP-10618 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: 2.4.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SingleNodeSetup.html is deprecated and not linked from the left side page. We should remove the document and use http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SingleCluster.html instead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10460) Please update on-line documentation for hadoop 2.3
[ https://issues.apache.org/jira/browse/HADOOP-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10460: --- Assignee: (was: Akira AJISAKA) Please update on-line documentation for hadoop 2.3 -- Key: HADOOP-10460 URL: https://issues.apache.org/jira/browse/HADOOP-10460 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 2.3.0 Environment: any Reporter: Darek Labels: newbie Attachments: HADOOP-10460.patch Documentation on page: http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/SingleNodeSetup.html contains steps like: $ cp conf/*.xml input but after checked out repository conf does not exists (I quess it was moved to etc/hadoop) Few lines below in section Execution there are steps: $ bin/hadoop namenode -format - OK $ bin/start-all.sh - this file has been removed -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10460) Please update on-line documentation for hadoop 2.3
[ https://issues.apache.org/jira/browse/HADOOP-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10460: --- Resolution: Invalid Status: Resolved (was: Patch Available) Filed HADOOP-10618 to remove the document. Closing this issue as invalid. If I misunderstood, feel free to reopen it. Please update on-line documentation for hadoop 2.3 -- Key: HADOOP-10460 URL: https://issues.apache.org/jira/browse/HADOOP-10460 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 2.3.0 Environment: any Reporter: Darek Assignee: Akira AJISAKA Labels: newbie Attachments: HADOOP-10460.patch Documentation on page: http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/SingleNodeSetup.html contains steps like: $ cp conf/*.xml input but after checked out repository conf does not exists (I quess it was moved to etc/hadoop) Few lines below in section Execution there are steps: $ bin/hadoop namenode -format - OK $ bin/start-all.sh - this file has been removed -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10618) Remove SingleNodeSetup.apt.vm
[ https://issues.apache.org/jira/browse/HADOOP-10618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10618: --- Attachment: HADOOP-10618.patch Attaching a patch. Remove SingleNodeSetup.apt.vm - Key: HADOOP-10618 URL: https://issues.apache.org/jira/browse/HADOOP-10618 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: 2.4.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Labels: newbie Attachments: HADOOP-10618.patch http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SingleNodeSetup.html is deprecated and not linked from the left side page. We should remove the document and use http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SingleCluster.html instead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10618) Remove SingleNodeSetup.apt.vm
[ https://issues.apache.org/jira/browse/HADOOP-10618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10618: --- Status: Patch Available (was: Open) Remove SingleNodeSetup.apt.vm - Key: HADOOP-10618 URL: https://issues.apache.org/jira/browse/HADOOP-10618 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: 2.4.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Labels: newbie Attachments: HADOOP-10618.patch http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SingleNodeSetup.html is deprecated and not linked from the left side page. We should remove the document and use http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SingleCluster.html instead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10618) Remove SingleNodeSetup.apt.vm
[ https://issues.apache.org/jira/browse/HADOOP-10618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10618: --- Attachment: HADOOP-10618.2.patch Thanks [~arpitagarwal] for the comment! Removed the content and added a link to SingleClusterSetup.html. I'm thinking it's better to remove the document itself in the next major release, so I wrote it to the doc. Remove SingleNodeSetup.apt.vm - Key: HADOOP-10618 URL: https://issues.apache.org/jira/browse/HADOOP-10618 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: 2.4.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Labels: newbie Attachments: HADOOP-10618.2.patch, HADOOP-10618.patch http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SingleNodeSetup.html is deprecated and not linked from the left side page. We should remove the document and use http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SingleCluster.html instead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10620) /docs/current doesn't point to the latest version 2.4.0
[ https://issues.apache.org/jira/browse/HADOOP-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14002882#comment-14002882 ] Akira AJISAKA commented on HADOOP-10620: Thanks [~jlaskowski] for the report. I sent a mail to common-dev@ for commiter's help. /docs/current doesn't point to the latest version 2.4.0 --- Key: HADOOP-10620 URL: https://issues.apache.org/jira/browse/HADOOP-10620 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: 2.4.0 Reporter: Jacek Laskowski http://hadoop.apache.org/docs/current/ points to 2.3.0 while 2.4.0's out. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10557: --- Attachment: HADOOP-10557.patch Attaching a patch. FsShell -cp -p does not preserve extended ACLs -- Key: HADOOP-10557 URL: https://issues.apache.org/jira/browse/HADOOP-10557 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Attachments: HADOOP-10557.patch This issue tracks enhancing FsShell cp to * preserve extended ACLs by -p option or * add a new command-line option for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10557: --- Affects Version/s: 2.4.0 Status: Patch Available (was: Open) FsShell -cp -p does not preserve extended ACLs -- Key: HADOOP-10557 URL: https://issues.apache.org/jira/browse/HADOOP-10557 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Attachments: HADOOP-10557.patch This issue tracks enhancing FsShell cp to * preserve extended ACLs by -p option or * add a new command-line option for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10602) Documentation has broken Go Back hyperlinks.
[ https://issues.apache.org/jira/browse/HADOOP-10602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10602: --- Attachment: HADOOP-10602.3.patch Updated the patch to remove 'Go Back' link from XAttr document. Documentation has broken Go Back hyperlinks. -- Key: HADOOP-10602 URL: https://issues.apache.org/jira/browse/HADOOP-10602 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 3.0.0, 2.4.0 Reporter: Chris Nauroth Assignee: Akira AJISAKA Priority: Trivial Labels: newbie Attachments: HADOOP-10602.2.patch, HADOOP-10602.3.patch, HADOOP-10602.patch Multiple pages of our documentation have Go Back links that are broken, because they point to an incorrect relative path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10557: --- Attachment: HADOOP-10557.2.patch Thanks [~jira.shegalov] for the comment! Updated the patch. FsShell -cp -p does not preserve extended ACLs -- Key: HADOOP-10557 URL: https://issues.apache.org/jira/browse/HADOOP-10557 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Attachments: HADOOP-10557.2.patch, HADOOP-10557.patch This issue tracks enhancing FsShell cp to * preserve extended ACLs by -p option or * add a new command-line option for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14025444#comment-14025444 ] Akira AJISAKA commented on HADOOP-8989: --- [~jonallen], sorry for late response. I'll review the patch by tomorrow. hadoop dfs -find feature Key: HADOOP-8989 URL: https://issues.apache.org/jira/browse/HADOOP-8989 Project: Hadoop Common Issue Type: New Feature Reporter: Marco Nicosia Assignee: Jonathan Allen Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch Both sysadmins and users make frequent use of the unix 'find' command, but Hadoop has no correlate. Without this, users are writing scripts which make heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs -lsr is somewhat taxing on the NameNode, and a really slow experience on the client side. Possibly an in-NameNode find operation would be only a bit more taxing on the NameNode, but significantly faster from the client's point of view? The minimum set of options I can think of which would make a Hadoop find command generally useful is (in priority order): * -type (file or directory, for now) * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) * -print0 (for piping to xargs -0) * -depth * -owner/-group (and -nouser/-nogroup) * -name (allowing for shell pattern, or even regex?) * -perm * -size One possible special case, but could possibly be really cool if it ran from within the NameNode: * -delete The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow. Lower priority, some people do use operators, mostly to execute -or searches such as: * find / \(-nouser -or -nogroup\) Finally, I thought I'd include a link to the [Posix spec for find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[
https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14025845#comment-14025845
]
Akira AJISAKA commented on HADOOP-8989:
---
Thanks for updating the patch. I compiled with the patch and tried some options.
1. When I executed -print0 option, newlines are included in the output as
follows:
{code}
# hdfs dfs -find /user/root -name '*.txt' -print0
abc.txt
def.txt
{code}
Newlines shouldn't be included in -print0 option.
2. In Find.java, spaces should be used instead of tabs.
{code}
// Initialize the static variables.
EXPRESSIONS = new Class[] {
// Operator Expressions
And.class,
// Action Expressions
Print.class,
// Navigation Expressions
// Matcher Expressions
Name.class
};
{code}
Minor nits:
3. In Expressions.java,
{code}
/**
* Returns the precendence of this expression
* (only applicable to operators).
*/
{code}
precendence should be precedence.
4. There are some trailing white spaces in Find.java.
5. In Print.java,
{code}
private Print(boolean appendNull) {
super();
setUsage(USAGE);
setHelp(HELP);
setAppendNull(appendNull);
}
private void setAppendNull(boolean appendNull) {
this.appendNull = appendNull;
}
{code}
can be simplified as
{code}
private Print(boolean appendNull) {
super();
setUsage(USAGE);
setHelp(HELP);
this.appendNull = appendNull;
}
{code}
6.
{code}
/**
* Construct a Print {@link Expression} with an operational ASCII NUL
* suffix.
*/
{code}
operational should be optional?
hadoop dfs -find feature
Key: HADOOP-8989
URL: https://issues.apache.org/jira/browse/HADOOP-8989
Project: Hadoop Common
Issue Type: New Feature
Reporter: Marco Nicosia
Assignee: Jonathan Allen
Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch
Both sysadmins and users make frequent use of the unix 'find' command, but
Hadoop has no correlate. Without this, users are writing scripts which make
heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs
-lsr is somewhat taxing on the NameNode, and a really slow experience on the
client side. Possibly an in-NameNode find operation would be only a bit more
taxing on the NameNode, but significantly faster from the client's point of
view?
The minimum set of options I can think of which would make a Hadoop find
command generally useful is (in priority order):
* -type (file or directory, for now)
* -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments)
* -print0 (for piping to xargs -0)
* -depth
* -owner/-group (and -nouser/-nogroup)
* -name (allowing for shell pattern, or even regex?)
* -perm
* -size
One possible special case, but could possibly be really cool if it ran from
within the NameNode:
* -delete
The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow.
Lower priority, some people do use operators, mostly to execute -or searches
such as:
* find / \(-nouser -or -nogroup\)
Finally, I thought I'd include a link to the [Posix spec for
find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html]
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[
https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10557:
---
Attachment: HADOOP-10557.3.patch
Thanks [~jira.shegalov] for the comment. Updated the patch to print a warning
as follows when src has some extended ACLs but dst doesn't support it.
{code}
# hdfs dfs -cp -p testfile file:///root/
14/06/10 08:28:12 WARN shell.CommandWithDestination: Extended ACLs
user:ajisakaa:rw-,group::r-- in hdfs://localhost:9000/user/root/testfile are
not preserved because the target FileSystem doesn't support ACLs.
{code}
FsShell -cp -p does not preserve extended ACLs
--
Key: HADOOP-10557
URL: https://issues.apache.org/jira/browse/HADOOP-10557
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch,
HADOOP-10557.patch
This issue tracks enhancing FsShell cp to
* preserve extended ACLs by -p option
or
* add a new command-line option for preserving extended ACLs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-6350) Documenting Hadoop metrics
[
https://issues.apache.org/jira/browse/HADOOP-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14027170#comment-14027170
]
Akira AJISAKA commented on HADOOP-6350:
---
Thanks [~arpitagarwal] for the review!
bq. I am not sure what this means - Each metrics record contains tags such as
ProcessName, SessionId, and Hostname as additional information along with
metrics.. How are these tags accessed, I don't see them in jconsole? Perhaps I
am missing some basic knowledge, let me know if so.
I can see them from jmx as follows:
{code}
name : Hadoop:service=NameNode,name=FSNamesystem,
modelerType : FSNamesystem,
tag.Context : dfs,
tag.HAState : active,
tag.Hostname : trunk,
MissingBlocks : 0,
ExpiredHeartbeats : 0,
...
{code}
Metrics records contain tags for grouping on host/queue/username etc.
{quote}
Namenode - snapshot metrics are missing.
DataNode - DataNodeInfo metrics are missing.
DataNode - FsDatasetState metrics are missing.
{quote}
These are not actually metrics. These are not collected or sinked by
{{MetricsSystem}}, so users cannot get them by file or ganglia. Users can get
these information only by jmx/jconsole.
bq. Nitpick: we should use title case consistently for sub-headings e.g.
rpcdetail -- RpcDetailed
The title shows the name of the Metrics, so it can be case inconsistent if the
name is inconsistent. For example, the name namenode is set by the following
code (NameNodeMetrics.java) :
{code}
final MetricsRegistry registry = new MetricsRegistry(namenode);
{code}
Documenting Hadoop metrics
--
Key: HADOOP-6350
URL: https://issues.apache.org/jira/browse/HADOOP-6350
Project: Hadoop Common
Issue Type: Improvement
Components: documentation, metrics
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Hong Tang
Assignee: Akira AJISAKA
Labels: metrics
Attachments: HADOOP-6350-sample-1.patch, HADOOP-6350-sample-2.patch,
HADOOP-6350-sample-3.patch, HADOOP-6350.4.patch, HADOOP-6350.5.patch,
HADOOP-6350.6.patch, HADOOP-6350.7.patch, sample1.png
Metrics should be part of public API, and should be clearly documented
similar to HADOOP-5073, so that we can reliably build tools on top of them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-6350) Documenting Hadoop metrics
[ https://issues.apache.org/jira/browse/HADOOP-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14027177#comment-14027177 ] Akira AJISAKA commented on HADOOP-6350: --- bq. These are not actually metrics. These are not collected or sinked by MetricsSystem, so users cannot get them by file or ganglia. Users can get these information only by jmx/jconsole. However, we should document these information also. I'll create a separate jira for tracking this. bq. As a separate discussion I think long term maintenance of this documentation will be challenging. I agree with you. If this document includes the information registered to MBeans (which can be accessed via jmx or jconsole), the maintenance will get more challenging. Documenting Hadoop metrics -- Key: HADOOP-6350 URL: https://issues.apache.org/jira/browse/HADOOP-6350 Project: Hadoop Common Issue Type: Improvement Components: documentation, metrics Affects Versions: 3.0.0, 2.1.0-beta Reporter: Hong Tang Assignee: Akira AJISAKA Labels: metrics Attachments: HADOOP-6350-sample-1.patch, HADOOP-6350-sample-2.patch, HADOOP-6350-sample-3.patch, HADOOP-6350.4.patch, HADOOP-6350.5.patch, HADOOP-6350.6.patch, HADOOP-6350.7.patch, sample1.png Metrics should be part of public API, and should be clearly documented similar to HADOOP-5073, so that we can reliably build tools on top of them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-6350) Documenting Hadoop metrics
[ https://issues.apache.org/jira/browse/HADOOP-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-6350: -- Attachment: HADOOP-6350.8.patch Updated the patch to fix a typo and remove trailing whitespaces. Documenting Hadoop metrics -- Key: HADOOP-6350 URL: https://issues.apache.org/jira/browse/HADOOP-6350 Project: Hadoop Common Issue Type: Improvement Components: documentation, metrics Affects Versions: 3.0.0, 2.1.0-beta Reporter: Hong Tang Assignee: Akira AJISAKA Labels: metrics Attachments: HADOOP-6350-sample-1.patch, HADOOP-6350-sample-2.patch, HADOOP-6350-sample-3.patch, HADOOP-6350.4.patch, HADOOP-6350.5.patch, HADOOP-6350.6.patch, HADOOP-6350.7.patch, HADOOP-6350.8.patch, sample1.png Metrics should be part of public API, and should be clearly documented similar to HADOOP-5073, so that we can reliably build tools on top of them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10680) Document metrics included in MXBean
Akira AJISAKA created HADOOP-10680:
--
Summary: Document metrics included in MXBean
Key: HADOOP-10680
URL: https://issues.apache.org/jira/browse/HADOOP-10680
Project: Hadoop Common
Issue Type: Improvement
Components: documentation
Reporter: Akira AJISAKA
Enhancement of HADOOP-6350.
For example, SnapshotInfo metrics in NameNode, DataNodeInfo and FsDatasetState
metrics in DataNode are not collected by {{MetricsSystem}} but registered to
{{MXBean}} via {{MBeanServer}}, so these metrics can be seen by jmx/jconsole.
These metrics should also be documented.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10682) Metrics are not output in trunk
Akira AJISAKA created HADOOP-10682:
--
Summary: Metrics are not output in trunk
Key: HADOOP-10682
URL: https://issues.apache.org/jira/browse/HADOOP-10682
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Reporter: Akira AJISAKA
Metrics are not output in trunk by the following configuration:
{code}
*sink.file.class=org.apache.Hadoop.metrics2.sink.FileSink
*.period=10
namenode.sink.file.filename=namenode-metrics.out
{code}
The below change worked well.
{code}
- namenode.sink.file.filename=namenode-metrics.out
+ NameNode.sink.file.filename=namenode-metrics.out
{code}
It means that an old configuration doesn't work on trunk. We should fix it or
document to use NameNode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14028436#comment-14028436 ] Akira AJISAKA commented on HADOOP-10557: Thanks [~cnauroth] for the comments! I wait for HADOOP-10561 to get committed. FsShell -cp -p does not preserve extended ACLs -- Key: HADOOP-10557 URL: https://issues.apache.org/jira/browse/HADOOP-10557 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch, HADOOP-10557.patch This issue tracks enhancing FsShell cp to * preserve extended ACLs by -p option or * add a new command-line option for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10557: --- Description: This issue tracks enhancing FsShell cp to add a new command-line option (-pa) for preserving extended ACLs. (was: This issue tracks enhancing FsShell cp to * preserve extended ACLs by -p option or * add a new command-line option for preserving extended ACLs.) FsShell -cp -p does not preserve extended ACLs -- Key: HADOOP-10557 URL: https://issues.apache.org/jira/browse/HADOOP-10557 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch, HADOOP-10557.patch This issue tracks enhancing FsShell cp to add a new command-line option (-pa) for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10682) Metrics are not output in trunk
[
https://issues.apache.org/jira/browse/HADOOP-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10682:
---
Description:
Metrics are not output in trunk by the following configuration:
{code}
*.sink.file.class=org.apache.Hadoop.metrics2.sink.FileSink
*.period=10
namenode.sink.file.filename=namenode-metrics.out
{code}
The below change worked well.
{code}
- namenode.sink.file.filename=namenode-metrics.out
+ NameNode.sink.file.filename=namenode-metrics.out
{code}
It means that an old configuration doesn't work on trunk. We should fix it or
document to use NameNode.
was:
Metrics are not output in trunk by the following configuration:
{code}
*sink.file.class=org.apache.Hadoop.metrics2.sink.FileSink
*.period=10
namenode.sink.file.filename=namenode-metrics.out
{code}
The below change worked well.
{code}
- namenode.sink.file.filename=namenode-metrics.out
+ NameNode.sink.file.filename=namenode-metrics.out
{code}
It means that an old configuration doesn't work on trunk. We should fix it or
document to use NameNode.
Metrics are not output in trunk
---
Key: HADOOP-10682
URL: https://issues.apache.org/jira/browse/HADOOP-10682
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Reporter: Akira AJISAKA
Metrics are not output in trunk by the following configuration:
{code}
*.sink.file.class=org.apache.Hadoop.metrics2.sink.FileSink
*.period=10
namenode.sink.file.filename=namenode-metrics.out
{code}
The below change worked well.
{code}
- namenode.sink.file.filename=namenode-metrics.out
+ NameNode.sink.file.filename=namenode-metrics.out
{code}
It means that an old configuration doesn't work on trunk. We should fix it or
document to use NameNode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-6350) Documenting Hadoop metrics
[ https://issues.apache.org/jira/browse/HADOOP-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14029977#comment-14029977 ] Akira AJISAKA commented on HADOOP-6350: --- I agree with you, I'll split the patch and create a separe jira. Thanks. Documenting Hadoop metrics -- Key: HADOOP-6350 URL: https://issues.apache.org/jira/browse/HADOOP-6350 Project: Hadoop Common Issue Type: Improvement Components: documentation, metrics Affects Versions: 3.0.0, 2.1.0-beta Reporter: Hong Tang Assignee: Akira AJISAKA Labels: metrics Attachments: HADOOP-6350-sample-1.patch, HADOOP-6350-sample-2.patch, HADOOP-6350-sample-3.patch, HADOOP-6350.4.patch, HADOOP-6350.5.patch, HADOOP-6350.6.patch, HADOOP-6350.7.patch, HADOOP-6350.8.patch, sample1.png Metrics should be part of public API, and should be clearly documented similar to HADOOP-5073, so that we can reliably build tools on top of them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-6350) Documenting Hadoop metrics
[ https://issues.apache.org/jira/browse/HADOOP-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-6350: -- Attachment: HADOOP-6350.9.patch Removed YARN/MR metrics from the v8 patch. Documenting Hadoop metrics -- Key: HADOOP-6350 URL: https://issues.apache.org/jira/browse/HADOOP-6350 Project: Hadoop Common Issue Type: Improvement Components: documentation, metrics Affects Versions: 3.0.0, 2.1.0-beta Reporter: Hong Tang Assignee: Akira AJISAKA Labels: metrics Attachments: HADOOP-6350-sample-1.patch, HADOOP-6350-sample-2.patch, HADOOP-6350-sample-3.patch, HADOOP-6350.4.patch, HADOOP-6350.5.patch, HADOOP-6350.6.patch, HADOOP-6350.7.patch, HADOOP-6350.8.patch, HADOOP-6350.9.patch, sample1.png Metrics should be part of public API, and should be clearly documented similar to HADOOP-5073, so that we can reliably build tools on top of them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-6350) Documenting Hadoop metrics
[ https://issues.apache.org/jira/browse/HADOOP-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-6350: -- Target Version/s: 2.5.0 (was: 3.0.0) Documenting Hadoop metrics -- Key: HADOOP-6350 URL: https://issues.apache.org/jira/browse/HADOOP-6350 Project: Hadoop Common Issue Type: Improvement Components: documentation, metrics Affects Versions: 3.0.0, 2.1.0-beta Reporter: Hong Tang Assignee: Akira AJISAKA Labels: metrics Attachments: HADOOP-6350-sample-1.patch, HADOOP-6350-sample-2.patch, HADOOP-6350-sample-3.patch, HADOOP-6350.4.patch, HADOOP-6350.5.patch, HADOOP-6350.6.patch, HADOOP-6350.7.patch, HADOOP-6350.8.patch, HADOOP-6350.9.patch, sample1.png Metrics should be part of public API, and should be clearly documented similar to HADOOP-5073, so that we can reliably build tools on top of them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10557: --- Attachment: HADOOP-10557.4.patch Updated the patch to add a new option (-pa). FsShell -cp -p does not preserve extended ACLs -- Key: HADOOP-10557 URL: https://issues.apache.org/jira/browse/HADOOP-10557 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch, HADOOP-10557.4.patch, HADOOP-10557.patch This issue tracks enhancing FsShell cp to add a new command-line option (-pa) for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10682) Metrics are not output in trunk
[
https://issues.apache.org/jira/browse/HADOOP-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA resolved HADOOP-10682.
Resolution: Not a Problem
It's caused by the change of HADOOP-10468. Regarding the issue, we should set
the property case sensitive from 2.5.0. Closing.
Metrics are not output in trunk
---
Key: HADOOP-10682
URL: https://issues.apache.org/jira/browse/HADOOP-10682
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Reporter: Akira AJISAKA
Metrics are not output in trunk by the following configuration:
{code}
*.sink.file.class=org.apache.Hadoop.metrics2.sink.FileSink
*.period=10
namenode.sink.file.filename=namenode-metrics.out
{code}
The below change worked well.
{code}
- namenode.sink.file.filename=namenode-metrics.out
+ NameNode.sink.file.filename=namenode-metrics.out
{code}
It means that an old configuration doesn't work on trunk. We should fix it or
document to use NameNode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10692) Update hadoop-metrics2.properties examples to be case sensitive
Akira AJISAKA created HADOOP-10692: -- Summary: Update hadoop-metrics2.properties examples to be case sensitive Key: HADOOP-10692 URL: https://issues.apache.org/jira/browse/HADOOP-10692 Project: Hadoop Common Issue Type: Bug Components: conf, metrics Affects Versions: 2.5.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA After HADOOP-10468, the prefix of the properties in metrics2 become case sensitive. We should also update hadoop-metrics2.properties examples to be case sensitive. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10692) Update hadoop-metrics2.properties examples to be case sensitive
[ https://issues.apache.org/jira/browse/HADOOP-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10692: --- Attachment: HADOOP-10692.patch Attaching a patch. Update hadoop-metrics2.properties examples to be case sensitive --- Key: HADOOP-10692 URL: https://issues.apache.org/jira/browse/HADOOP-10692 Project: Hadoop Common Issue Type: Bug Components: conf, metrics Affects Versions: 2.5.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Labels: newbie Attachments: HADOOP-10692.patch After HADOOP-10468, the prefix of the properties in metrics2 become case sensitive. We should also update hadoop-metrics2.properties examples to be case sensitive. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10692) Update hadoop-metrics2.properties examples to be case sensitive
[ https://issues.apache.org/jira/browse/HADOOP-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10692: --- Target Version/s: 2.5.0 Status: Patch Available (was: Open) Update hadoop-metrics2.properties examples to be case sensitive --- Key: HADOOP-10692 URL: https://issues.apache.org/jira/browse/HADOOP-10692 Project: Hadoop Common Issue Type: Bug Components: conf, metrics Affects Versions: 2.5.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Labels: newbie Attachments: HADOOP-10692.patch After HADOOP-10468, the prefix of the properties in metrics2 become case sensitive. We should also update hadoop-metrics2.properties examples to be case sensitive. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10692) Update hadoop-metrics2.properties examples to be case sensitive
[ https://issues.apache.org/jira/browse/HADOOP-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10692: --- Attachment: HADOOP-10692.2.patch Updated metrics2 package-info also. Update hadoop-metrics2.properties examples to be case sensitive --- Key: HADOOP-10692 URL: https://issues.apache.org/jira/browse/HADOOP-10692 Project: Hadoop Common Issue Type: Bug Components: conf, metrics Affects Versions: 2.5.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Labels: newbie Attachments: HADOOP-10692.2.patch, HADOOP-10692.patch After HADOOP-10468, the prefix of the properties in metrics2 become case sensitive. We should also update hadoop-metrics2.properties examples to be case sensitive. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10692) Update metrics2 document and examples to be case sensitive
[ https://issues.apache.org/jira/browse/HADOOP-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10692: --- Description: After HADOOP-10468, the prefix of the properties in metrics2 become case sensitive. We should also update package-info and hadoop-metrics2.properties examples to be case sensitive. (was: After HADOOP-10468, the prefix of the properties in metrics2 become case sensitive. We should also update hadoop-metrics2.properties examples to be case sensitive.) Summary: Update metrics2 document and examples to be case sensitive (was: Update hadoop-metrics2.properties examples to be case sensitive) Update metrics2 document and examples to be case sensitive -- Key: HADOOP-10692 URL: https://issues.apache.org/jira/browse/HADOOP-10692 Project: Hadoop Common Issue Type: Bug Components: conf, metrics Affects Versions: 2.5.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Labels: newbie Attachments: HADOOP-10692.2.patch, HADOOP-10692.patch After HADOOP-10468, the prefix of the properties in metrics2 become case sensitive. We should also update package-info and hadoop-metrics2.properties examples to be case sensitive. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-6350) Documenting Hadoop metrics
[ https://issues.apache.org/jira/browse/HADOOP-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14030250#comment-14030250 ] Akira AJISAKA commented on HADOOP-6350: --- Thanks [~arpitagarwal]! Documenting Hadoop metrics -- Key: HADOOP-6350 URL: https://issues.apache.org/jira/browse/HADOOP-6350 Project: Hadoop Common Issue Type: Improvement Components: documentation, metrics Affects Versions: 3.0.0, 2.1.0-beta Reporter: Hong Tang Assignee: Akira AJISAKA Labels: metrics Fix For: 3.0.0, 2.5.0 Attachments: HADOOP-6350-sample-1.patch, HADOOP-6350-sample-2.patch, HADOOP-6350-sample-3.patch, HADOOP-6350.4.patch, HADOOP-6350.5.patch, HADOOP-6350.6.patch, HADOOP-6350.7.patch, HADOOP-6350.8.patch, HADOOP-6350.9.patch, sample1.png Metrics should be part of public API, and should be clearly documented similar to HADOOP-5073, so that we can reliably build tools on top of them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10692) Update metrics2 document and examples to be case sensitive
[ https://issues.apache.org/jira/browse/HADOOP-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14030914#comment-14030914 ] Akira AJISAKA commented on HADOOP-10692: The patch is just to update the document and examples, so new test are not needed. Update metrics2 document and examples to be case sensitive -- Key: HADOOP-10692 URL: https://issues.apache.org/jira/browse/HADOOP-10692 Project: Hadoop Common Issue Type: Bug Components: conf, metrics Affects Versions: 2.5.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Labels: newbie Attachments: HADOOP-10692.2.patch, HADOOP-10692.patch After HADOOP-10468, the prefix of the properties in metrics2 become case sensitive. We should also update package-info and hadoop-metrics2.properties examples to be case sensitive. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10557: --- Attachment: HADOOP-10557.5.patch Thanks [~cnauroth] for the comment! Updated the patch. FsShell -cp -p does not preserve extended ACLs -- Key: HADOOP-10557 URL: https://issues.apache.org/jira/browse/HADOOP-10557 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch, HADOOP-10557.4.patch, HADOOP-10557.5.patch, HADOOP-10557.patch This issue tracks enhancing FsShell cp to add a new command-line option (-pa) for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10557) FsShell -cp -pa option for preserving extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034054#comment-14034054 ] Akira AJISAKA commented on HADOOP-10557: Thanks [~cnauroth] for testing! FsShell -cp -pa option for preserving extended ACLs --- Key: HADOOP-10557 URL: https://issues.apache.org/jira/browse/HADOOP-10557 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 2.4.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Fix For: 3.0.0, 2.5.0 Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch, HADOOP-10557.4.patch, HADOOP-10557.5.patch, HADOOP-10557.patch This issue tracks enhancing FsShell cp to add a new command-line option (-pa) for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034059#comment-14034059
]
Akira AJISAKA commented on HADOOP-9705:
---
I'll rebase the patch and make sure it works with all the attributes by
tomorrow. Thanks!
FsShell cp -p does not preserve directory attibutes
---
Key: HADOOP-9705
URL: https://issues.apache.org/jira/browse/HADOOP-9705
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Stephen Chu
Assignee: Akira AJISAKA
Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
HADOOP-9705.4.patch, HADOOP-9705.patch
HADOOP-9338 added the -p flag to preserve file attributes when copying.
However, cp -p does not preserve directory attributes. It'd be useful to add
this functionality.
For example, the following shows that the modified time is not preserved
{code}
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
/user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
Found 1 items
drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
/user/schu/testDir1 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
Found 2 items
drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$
{code}
The preserve logic is in CommandWithDestination#copyFileToTarget, which is
only called with files.
{code}
protected void processPath(PathData src, PathData dst) throws IOException {
if (src.stat.isSymlink()) {
// TODO: remove when FileContext is supported, this needs to either
// copy the symlink or deref the symlink
throw new PathOperationException(src.toString());
} else if (src.stat.isFile()) {
copyFileToTarget(src, dst);
} else if (src.stat.isDirectory() !isRecursive()) {
throw new PathIsDirectoryException(src.toString());
}
}
{code}
{code}
/**
* Copies the source file to the target.
* @param src item to copy
* @param target where to copy the item
* @throws IOException if copy fails
*/
protected void copyFileToTarget(PathData src, PathData target) throws
IOException {
src.fs.setVerifyChecksum(verifyChecksum);
if (src != null) {
throw new PathExistsException(hi);
}
InputStream in = null;
try {
in = src.fs.open(src.path);
copyStreamToTarget(in, target);
if(preserve) {
target.fs.setTimes(
target.path,
src.stat.getModificationTime(),
src.stat.getAccessTime());
target.fs.setOwner(
target.path,
src.stat.getOwner(),
src.stat.getGroup());
target.fs.setPermission(
target.path,
src.stat.getPermission());
System.out.println(Preserving);
if (src.fs.equals(target.fs)) {
System.out.println(Same filesystems);
src.fs.preserveAttributes(src.path, target.path);
}
throw new IOException(hi);
}
} finally {
IOUtils.closeStream(in);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-9705:
--
Attachment: HADOOP-9705.5.patch
Rebased the patch.
FsShell cp -p does not preserve directory attibutes
---
Key: HADOOP-9705
URL: https://issues.apache.org/jira/browse/HADOOP-9705
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Stephen Chu
Assignee: Akira AJISAKA
Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
HADOOP-9705.4.patch, HADOOP-9705.5.patch, HADOOP-9705.patch
HADOOP-9338 added the -p flag to preserve file attributes when copying.
However, cp -p does not preserve directory attributes. It'd be useful to add
this functionality.
For example, the following shows that the modified time is not preserved
{code}
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
/user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
Found 1 items
drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
/user/schu/testDir1 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
Found 2 items
drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$
{code}
The preserve logic is in CommandWithDestination#copyFileToTarget, which is
only called with files.
{code}
protected void processPath(PathData src, PathData dst) throws IOException {
if (src.stat.isSymlink()) {
// TODO: remove when FileContext is supported, this needs to either
// copy the symlink or deref the symlink
throw new PathOperationException(src.toString());
} else if (src.stat.isFile()) {
copyFileToTarget(src, dst);
} else if (src.stat.isDirectory() !isRecursive()) {
throw new PathIsDirectoryException(src.toString());
}
}
{code}
{code}
/**
* Copies the source file to the target.
* @param src item to copy
* @param target where to copy the item
* @throws IOException if copy fails
*/
protected void copyFileToTarget(PathData src, PathData target) throws
IOException {
src.fs.setVerifyChecksum(verifyChecksum);
if (src != null) {
throw new PathExistsException(hi);
}
InputStream in = null;
try {
in = src.fs.open(src.path);
copyStreamToTarget(in, target);
if(preserve) {
target.fs.setTimes(
target.path,
src.stat.getModificationTime(),
src.stat.getAccessTime());
target.fs.setOwner(
target.path,
src.stat.getOwner(),
src.stat.getGroup());
target.fs.setPermission(
target.path,
src.stat.getPermission());
System.out.println(Preserving);
if (src.fs.equals(target.fs)) {
System.out.println(Same filesystems);
src.fs.preserveAttributes(src.path, target.path);
}
throw new IOException(hi);
}
} finally {
IOUtils.closeStream(in);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[
https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034491#comment-14034491
]
Akira AJISAKA commented on HADOOP-8989:
---
Thanks for updating the patch! Mostly looks good. I verified -print0 option and
the following command worked well.
{code}
# hdfs dfs -find /user/root/ -name file* -print0 | xargs -0 hdfs dfs -rm
{code}
Minor comment:
{code}
private static String[] HELP =
{ Finds all files that match the specified expression and
+ applies selected actions to them. };
{code}
Would you change the code like other commands? i.e.
{code}
private static String[] HELP =
{ Finds all files that match the specified expression and,
applies selected actions to them. };
{code}
In addition, can you please include a document in the patch for keeping
consistency between source code and document? (i.e. Would you split
HADOOP-10580?)
[~umamaheswararao], do you have any comments on this patch? I'll appreciate
your reviews.
hadoop dfs -find feature
Key: HADOOP-8989
URL: https://issues.apache.org/jira/browse/HADOOP-8989
Project: Hadoop Common
Issue Type: New Feature
Reporter: Marco Nicosia
Assignee: Jonathan Allen
Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
HADOOP-8989.patch
Both sysadmins and users make frequent use of the unix 'find' command, but
Hadoop has no correlate. Without this, users are writing scripts which make
heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs
-lsr is somewhat taxing on the NameNode, and a really slow experience on the
client side. Possibly an in-NameNode find operation would be only a bit more
taxing on the NameNode, but significantly faster from the client's point of
view?
The minimum set of options I can think of which would make a Hadoop find
command generally useful is (in priority order):
* -type (file or directory, for now)
* -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments)
* -print0 (for piping to xargs -0)
* -depth
* -owner/-group (and -nouser/-nogroup)
* -name (allowing for shell pattern, or even regex?)
* -perm
* -size
One possible special case, but could possibly be really cool if it ran from
within the NameNode:
* -delete
The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow.
Lower priority, some people do use operators, mostly to execute -or searches
such as:
* find / \(-nouser -or -nogroup\)
Finally, I thought I'd include a link to the [Posix spec for
find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html]
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10059) RPC authentication and authorization metrics overflow to negative values on busy clusters
[ https://issues.apache.org/jira/browse/HADOOP-10059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10059: --- Target Version/s: 2.5.0 RPC authentication and authorization metrics overflow to negative values on busy clusters - Key: HADOOP-10059 URL: https://issues.apache.org/jira/browse/HADOOP-10059 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: 0.23.9, 2.2.0 Reporter: Jason Lowe Assignee: Tsuyoshi OZAWA Priority: Minor Attachments: HADOOP-10059.1.patch, HADOOP-10059.2.patch The RPC metrics for authorization and authentication successes can easily overflow to negative values on a busy cluster that has been up for a long time. We should consider providing 64-bit values for these counters. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HADOOP-10065) Fix namenode format documentation
[
https://issues.apache.org/jira/browse/HADOOP-10065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA reassigned HADOOP-10065:
--
Assignee: Akira AJISAKA (was: Arpit Gupta)
Fix namenode format documentation
-
Key: HADOOP-10065
URL: https://issues.apache.org/jira/browse/HADOOP-10065
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Affects Versions: 2.2.0
Reporter: Arpit Gupta
Assignee: Akira AJISAKA
Priority: Minor
Attachments: HADOOP-10065.2.patch, HADOOP-10065.patch
Current namenode format doc
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CommandsManual.html#namenode
Does not list the various options format can be called with and their use.
{code}
[-format [-clusterid cid ] [-force] [-nonInteractive] ]
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10065) Fix namenode format documentation
[
https://issues.apache.org/jira/browse/HADOOP-10065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10065:
---
Attachment: HADOOP-10065.3.patch
Attaching a patch to fix formatting and remove trailing whitespace.
Fix namenode format documentation
-
Key: HADOOP-10065
URL: https://issues.apache.org/jira/browse/HADOOP-10065
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Affects Versions: 2.2.0
Reporter: Arpit Gupta
Assignee: Akira AJISAKA
Priority: Minor
Attachments: HADOOP-10065.2.patch, HADOOP-10065.3.patch,
HADOOP-10065.patch
Current namenode format doc
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CommandsManual.html#namenode
Does not list the various options format can be called with and their use.
{code}
[-format [-clusterid cid ] [-force] [-nonInteractive] ]
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10065) Fix namenode format documentation
[
https://issues.apache.org/jira/browse/HADOOP-10065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041427#comment-14041427
]
Akira AJISAKA commented on HADOOP-10065:
Hi [~arpitgupta], what's your status of this issue?
I assigned this issue to me, but feel free to reassign it.
Fix namenode format documentation
-
Key: HADOOP-10065
URL: https://issues.apache.org/jira/browse/HADOOP-10065
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Affects Versions: 2.2.0
Reporter: Arpit Gupta
Assignee: Akira AJISAKA
Priority: Minor
Attachments: HADOOP-10065.2.patch, HADOOP-10065.3.patch,
HADOOP-10065.patch
Current namenode format doc
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CommandsManual.html#namenode
Does not list the various options format can be called with and their use.
{code}
[-format [-clusterid cid ] [-force] [-nonInteractive] ]
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10065) Fix namenode format documentation
[
https://issues.apache.org/jira/browse/HADOOP-10065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10065:
---
Target Version/s: 2.5.0 (was: 2.3.0)
Fix namenode format documentation
-
Key: HADOOP-10065
URL: https://issues.apache.org/jira/browse/HADOOP-10065
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Affects Versions: 2.2.0
Reporter: Arpit Gupta
Assignee: Akira AJISAKA
Priority: Minor
Attachments: HADOOP-10065.2.patch, HADOOP-10065.3.patch,
HADOOP-10065.patch
Current namenode format doc
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CommandsManual.html#namenode
Does not list the various options format can be called with and their use.
{code}
[-format [-clusterid cid ] [-force] [-nonInteractive] ]
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10065) Fix namenode format documentation
[
https://issues.apache.org/jira/browse/HADOOP-10065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041436#comment-14041436
]
Akira AJISAKA commented on HADOOP-10065:
Thanks [~arpitgupta]!
Fix namenode format documentation
-
Key: HADOOP-10065
URL: https://issues.apache.org/jira/browse/HADOOP-10065
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Affects Versions: 2.2.0
Reporter: Arpit Gupta
Assignee: Akira AJISAKA
Priority: Minor
Attachments: HADOOP-10065.2.patch, HADOOP-10065.3.patch,
HADOOP-10065.patch
Current namenode format doc
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CommandsManual.html#namenode
Does not list the various options format can be called with and their use.
{code}
[-format [-clusterid cid ] [-force] [-nonInteractive] ]
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10119) Document hadoop archive -p option
[ https://issues.apache.org/jira/browse/HADOOP-10119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10119: --- Target Version/s: 2.5.0 (was: ) Document hadoop archive -p option - Key: HADOOP-10119 URL: https://issues.apache.org/jira/browse/HADOOP-10119 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 2.2.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Labels: newbie Attachments: HADOOP-10119.patch Now hadoop archive -p (relative parent path) option is required but the option is not documented. See http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CommandsManual.html#archive . -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10121) Fix javadoc spelling for HadoopArchives#writeTopLevelDirs
[
https://issues.apache.org/jira/browse/HADOOP-10121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10121:
---
Priority: Trivial (was: Minor)
Target Version/s: 2.5.0 (was: )
Fix javadoc spelling for HadoopArchives#writeTopLevelDirs
-
Key: HADOOP-10121
URL: https://issues.apache.org/jira/browse/HADOOP-10121
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Affects Versions: 2.2.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Trivial
Labels: newbie
Attachments: HADOOP-10121.patch
There's a misspelling at HadoopArchives.java. It should be fixed as follows:
{code}
- * @param parentPath the parent path that you wnat the archives
+ * @param parentPath the parent path that you want the archives
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10231) Add some components in Native Libraries document
[ https://issues.apache.org/jira/browse/HADOOP-10231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10231: --- Target Version/s: 2.5.0 (was: ) Add some components in Native Libraries document Key: HADOOP-10231 URL: https://issues.apache.org/jira/browse/HADOOP-10231 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: 2.2.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Labels: newbie Attachments: HADOOP-10231.patch The documented components in Native Libraries are only zlib and gzip. Now Native Libraries includes some other components such as other compression formats (lz4, snappy), libhdfs and fuse module. These components should be documented. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10387) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/HADOOP-10387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10387: --- Hadoop Flags: (was: Reviewed) Misspelling of threshold in log4j.properties for tests -- Key: HADOOP-10387 URL: https://issues.apache.org/jira/browse/HADOOP-10387 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 2.3.0 Reporter: Kenji Kikushima Priority: Minor Attachments: HADOOP-10387.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10387) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/HADOOP-10387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041452#comment-14041452 ] Akira AJISAKA commented on HADOOP-10387: Hi [~kj-ki], now hadoop-azure and hadoop-nfs projects have typo 'threshhold', so would you please update the patch to fix them? Misspelling of threshold in log4j.properties for tests -- Key: HADOOP-10387 URL: https://issues.apache.org/jira/browse/HADOOP-10387 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 2.3.0 Reporter: Kenji Kikushima Priority: Minor Attachments: HADOOP-10387.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10387) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/HADOOP-10387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10387: --- Component/s: conf Target Version/s: 2.5.0 Misspelling of threshold in log4j.properties for tests -- Key: HADOOP-10387 URL: https://issues.apache.org/jira/browse/HADOOP-10387 Project: Hadoop Common Issue Type: Bug Components: conf, test Affects Versions: 2.3.0 Reporter: Kenji Kikushima Priority: Minor Attachments: HADOOP-10387-2.patch, HADOOP-10387.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10387) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/HADOOP-10387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14042437#comment-14042437 ] Akira AJISAKA commented on HADOOP-10387: +1 (non-binding), pending Jenkins. Misspelling of threshold in log4j.properties for tests -- Key: HADOOP-10387 URL: https://issues.apache.org/jira/browse/HADOOP-10387 Project: Hadoop Common Issue Type: Bug Components: conf, test Affects Versions: 2.3.0 Reporter: Kenji Kikushima Priority: Minor Attachments: HADOOP-10387-2.patch, HADOOP-10387.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14045271#comment-14045271
]
Akira AJISAKA commented on HADOOP-9705:
---
Thanks Chris for reviewing!
FsShell cp -p does not preserve directory attibutes
---
Key: HADOOP-9705
URL: https://issues.apache.org/jira/browse/HADOOP-9705
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Stephen Chu
Assignee: Akira AJISAKA
Fix For: 3.0.0, 2.5.0
Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
HADOOP-9705.4.patch, HADOOP-9705.5.patch, HADOOP-9705.patch
HADOOP-9338 added the -p flag to preserve file attributes when copying.
However, cp -p does not preserve directory attributes. It'd be useful to add
this functionality.
For example, the following shows that the modified time is not preserved
{code}
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
/user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
Found 1 items
drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
/user/schu/testDir1 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
Found 2 items
drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
[schu@hdfs-snapshots-1 ~]$
{code}
The preserve logic is in CommandWithDestination#copyFileToTarget, which is
only called with files.
{code}
protected void processPath(PathData src, PathData dst) throws IOException {
if (src.stat.isSymlink()) {
// TODO: remove when FileContext is supported, this needs to either
// copy the symlink or deref the symlink
throw new PathOperationException(src.toString());
} else if (src.stat.isFile()) {
copyFileToTarget(src, dst);
} else if (src.stat.isDirectory() !isRecursive()) {
throw new PathIsDirectoryException(src.toString());
}
}
{code}
{code}
/**
* Copies the source file to the target.
* @param src item to copy
* @param target where to copy the item
* @throws IOException if copy fails
*/
protected void copyFileToTarget(PathData src, PathData target) throws
IOException {
src.fs.setVerifyChecksum(verifyChecksum);
if (src != null) {
throw new PathExistsException(hi);
}
InputStream in = null;
try {
in = src.fs.open(src.path);
copyStreamToTarget(in, target);
if(preserve) {
target.fs.setTimes(
target.path,
src.stat.getModificationTime(),
src.stat.getAccessTime());
target.fs.setOwner(
target.path,
src.stat.getOwner(),
src.stat.getGroup());
target.fs.setPermission(
target.path,
src.stat.getPermission());
System.out.println(Preserving);
if (src.fs.equals(target.fs)) {
System.out.println(Same filesystems);
src.fs.preserveAttributes(src.path, target.path);
}
throw new IOException(hi);
}
} finally {
IOUtils.closeStream(in);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-8808) Update FsShell documentation to mention deprecation of some of the commands, and mention alternatives
[ https://issues.apache.org/jira/browse/HADOOP-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14052178#comment-14052178 ] Akira AJISAKA commented on HADOOP-8808: --- The test failure is not related to the patch. It was reported at HADOOP-10406. Update FsShell documentation to mention deprecation of some of the commands, and mention alternatives - Key: HADOOP-8808 URL: https://issues.apache.org/jira/browse/HADOOP-8808 Project: Hadoop Common Issue Type: Bug Components: documentation, fs Affects Versions: 2.2.0 Reporter: Hemanth Yamijala Assignee: Akira AJISAKA Attachments: HADOOP-8808.2.patch, HADOOP-8808.patch In HADOOP-7286, we deprecated the following 3 commands dus, lsr and rmr, in favour of du -s, ls -r and rm -r respectively. The FsShell documentation should be updated to mention these, so that users can start switching. Also, there are places where we refer to the deprecated commands as alternatives. This can be changed as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10468) TestMetricsSystemImpl.testMultiThreadedPublish fails intermediately
[
https://issues.apache.org/jira/browse/HADOOP-10468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14052185#comment-14052185
]
Akira AJISAKA commented on HADOOP-10468:
bq. I'm hoping there's a way we can fix the underlying issue without breaking
existing metrics2 property files
I agree with you, [~jlowe]. I'm trying to find the way.
TestMetricsSystemImpl.testMultiThreadedPublish fails intermediately
---
Key: HADOOP-10468
URL: https://issues.apache.org/jira/browse/HADOOP-10468
Project: Hadoop Common
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Fix For: 2.5.0
Attachments: HADOOP-10468.000.patch, HADOOP-10468.001.patch
{{TestMetricsSystemImpl.testMultiThreadedPublish}} can fail intermediately
due to the insufficient size of the sink queue:
{code}
2014-04-06 21:34:55,269 WARN impl.MetricsSinkAdapter
(MetricsSinkAdapter.java:putMetricsImmediate(107)) - Collector has a full
queue and can't consume the given metrics.
2014-04-06 21:34:55,270 WARN impl.MetricsSinkAdapter
(MetricsSinkAdapter.java:putMetricsImmediate(107)) - Collector has a full
queue and can't consume the given metrics.
2014-04-06 21:34:55,271 WARN impl.MetricsSinkAdapter
(MetricsSinkAdapter.java:putMetricsImmediate(107)) - Collector has a full
queue and can't consume the given metrics.
{code}
The unit test should increase the default queue size to avoid intermediate
failure.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10468) TestMetricsSystemImpl.testMultiThreadedPublish fails intermediately
[
https://issues.apache.org/jira/browse/HADOOP-10468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14052200#comment-14052200
]
Akira AJISAKA commented on HADOOP-10468:
Attaching a patch to revert HADOOP-10468 and make 'Collector' to lower case.
{code}
- .add(Test.sink.Collector. + MetricsConfig.QUEUE_CAPACITY_KEY,
+ .add(test.sink.collector. + MetricsConfig.QUEUE_CAPACITY_KEY,
{code}
{code}
-ms.registerSink(Collector,
+ms.registerSink(collector,
{code}
Using debugger, I confirmed the queue capacity of the sink was set to 10.
TestMetricsSystemImpl.testMultiThreadedPublish fails intermediately
---
Key: HADOOP-10468
URL: https://issues.apache.org/jira/browse/HADOOP-10468
Project: Hadoop Common
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Fix For: 2.5.0
Attachments: HADOOP-10468.000.patch, HADOOP-10468.001.patch
{{TestMetricsSystemImpl.testMultiThreadedPublish}} can fail intermediately
due to the insufficient size of the sink queue:
{code}
2014-04-06 21:34:55,269 WARN impl.MetricsSinkAdapter
(MetricsSinkAdapter.java:putMetricsImmediate(107)) - Collector has a full
queue and can't consume the given metrics.
2014-04-06 21:34:55,270 WARN impl.MetricsSinkAdapter
(MetricsSinkAdapter.java:putMetricsImmediate(107)) - Collector has a full
queue and can't consume the given metrics.
2014-04-06 21:34:55,271 WARN impl.MetricsSinkAdapter
(MetricsSinkAdapter.java:putMetricsImmediate(107)) - Collector has a full
queue and can't consume the given metrics.
{code}
The unit test should increase the default queue size to avoid intermediate
failure.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10468) TestMetricsSystemImpl.testMultiThreadedPublish fails intermediately
[
https://issues.apache.org/jira/browse/HADOOP-10468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10468:
---
Attachment: HADOOP-10468.2.patch
TestMetricsSystemImpl.testMultiThreadedPublish fails intermediately
---
Key: HADOOP-10468
URL: https://issues.apache.org/jira/browse/HADOOP-10468
Project: Hadoop Common
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Fix For: 2.5.0
Attachments: HADOOP-10468.000.patch, HADOOP-10468.001.patch,
HADOOP-10468.2.patch
{{TestMetricsSystemImpl.testMultiThreadedPublish}} can fail intermediately
due to the insufficient size of the sink queue:
{code}
2014-04-06 21:34:55,269 WARN impl.MetricsSinkAdapter
(MetricsSinkAdapter.java:putMetricsImmediate(107)) - Collector has a full
queue and can't consume the given metrics.
2014-04-06 21:34:55,270 WARN impl.MetricsSinkAdapter
(MetricsSinkAdapter.java:putMetricsImmediate(107)) - Collector has a full
queue and can't consume the given metrics.
2014-04-06 21:34:55,271 WARN impl.MetricsSinkAdapter
(MetricsSinkAdapter.java:putMetricsImmediate(107)) - Collector has a full
queue and can't consume the given metrics.
{code}
The unit test should increase the default queue size to avoid intermediate
failure.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10468) TestMetricsSystemImpl.testMultiThreadedPublish fails intermediately
[
https://issues.apache.org/jira/browse/HADOOP-10468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-10468:
---
Affects Version/s: 2.5.0
Status: Patch Available (was: Reopened)
TestMetricsSystemImpl.testMultiThreadedPublish fails intermediately
---
Key: HADOOP-10468
URL: https://issues.apache.org/jira/browse/HADOOP-10468
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Haohui Mai
Assignee: Haohui Mai
Fix For: 2.5.0
Attachments: HADOOP-10468.000.patch, HADOOP-10468.001.patch,
HADOOP-10468.2.patch
{{TestMetricsSystemImpl.testMultiThreadedPublish}} can fail intermediately
due to the insufficient size of the sink queue:
{code}
2014-04-06 21:34:55,269 WARN impl.MetricsSinkAdapter
(MetricsSinkAdapter.java:putMetricsImmediate(107)) - Collector has a full
queue and can't consume the given metrics.
2014-04-06 21:34:55,270 WARN impl.MetricsSinkAdapter
(MetricsSinkAdapter.java:putMetricsImmediate(107)) - Collector has a full
queue and can't consume the given metrics.
2014-04-06 21:34:55,271 WARN impl.MetricsSinkAdapter
(MetricsSinkAdapter.java:putMetricsImmediate(107)) - Collector has a full
queue and can't consume the given metrics.
{code}
The unit test should increase the default queue size to avoid intermediate
failure.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
