[jira] [Updated] (HDFS-8581) ContentSummary on / skips further counts on yielding lock
[ https://issues.apache.org/jira/browse/HDFS-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-8581: - Fix Version/s: 2.8.0 > ContentSummary on / skips further counts on yielding lock > - > > Key: HDFS-8581 > URL: https://issues.apache.org/jira/browse/HDFS-8581 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: tongshiquan >Assignee: J.Andreina >Priority: Minor > Fix For: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 > > Attachments: HDFS-8581.1.patch, HDFS-8581.2.patch, HDFS-8581.3.patch, > HDFS-8581.4.patch > > > If one directory such as "/result" exists about 20 files, then when > execute "hdfs dfs -count /", the result will go wrong. For all directories > whose name after "/result", file num will not be included. > My cluster see as below, "/result_1433858936" is the directory exist huge > files, and files in "/sparkJobHistory", "/tmp", "/user" are not included > vm-221:/export1/BigData/current # hdfs dfs -ls / > 15/06/11 11:00:17 INFO hdfs.PeerCache: SocketCache disabled. > Found 9 items > -rw-r--r-- 3 hdfs supergroup 0 2015-06-08 12:10 > /PRE_CREATE_DIR.SUCCESS > drwxr-x--- - flume hadoop 0 2015-06-08 12:08 /flume > drwx-- - hbase hadoop 0 2015-06-10 15:25 /hbase > drwxr-xr-x - hdfs supergroup 0 2015-06-10 17:19 /hyt > drwxrwxrwx - mapred hadoop 0 2015-06-08 12:08 /mr-history > drwxr-xr-x - hdfs supergroup 0 2015-06-09 22:10 > /result_1433858936 > drwxrwxrwx - spark supergroup 0 2015-06-10 19:15 /sparkJobHistory > drwxrwxrwx - hdfs hadoop 0 2015-06-08 12:14 /tmp > drwxrwxrwx - hdfs hadoop 0 2015-06-09 21:57 /user > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count / > 15/06/11 11:00:24 INFO hdfs.PeerCache: SocketCache disabled. > 1043 171536 1756375688 / > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /PRE_CREATE_DIR.SUCCESS > 15/06/11 11:00:30 INFO hdfs.PeerCache: SocketCache disabled. >01 0 /PRE_CREATE_DIR.SUCCESS > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /flume > 15/06/11 11:00:41 INFO hdfs.PeerCache: SocketCache disabled. >10 0 /flume > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /hbase > 15/06/11 11:00:49 INFO hdfs.PeerCache: SocketCache disabled. > 36 18 14807 /hbase > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /hyt > 15/06/11 11:01:09 INFO hdfs.PeerCache: SocketCache disabled. >10 0 /hyt > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /mr-history > 15/06/11 11:01:18 INFO hdfs.PeerCache: SocketCache disabled. >30 0 /mr-history > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /result_1433858936 > 15/06/11 11:01:29 INFO hdfs.PeerCache: SocketCache disabled. > 1001 171517 1756360881 /result_1433858936 > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /sparkJobHistory > 15/06/11 11:01:41 INFO hdfs.PeerCache: SocketCache disabled. >13 21785 /sparkJobHistory > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /tmp > 15/06/11 11:01:48 INFO hdfs.PeerCache: SocketCache disabled. > 176 35958 /tmp > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /user > 15/06/11 11:01:55 INFO hdfs.PeerCache: SocketCache disabled. > 121 19077 /user -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8581) ContentSummary on / skips further counts on yielding lock
[ https://issues.apache.org/jira/browse/HDFS-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8581: - Fix Version/s: (was: 2.8.0) 2.6.5 2.7.3 > ContentSummary on / skips further counts on yielding lock > - > > Key: HDFS-8581 > URL: https://issues.apache.org/jira/browse/HDFS-8581 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: tongshiquan >Assignee: J.Andreina >Priority: Minor > Fix For: 2.7.3, 2.6.5 > > Attachments: HDFS-8581.1.patch, HDFS-8581.2.patch, HDFS-8581.3.patch, > HDFS-8581.4.patch > > > If one directory such as "/result" exists about 20 files, then when > execute "hdfs dfs -count /", the result will go wrong. For all directories > whose name after "/result", file num will not be included. > My cluster see as below, "/result_1433858936" is the directory exist huge > files, and files in "/sparkJobHistory", "/tmp", "/user" are not included > vm-221:/export1/BigData/current # hdfs dfs -ls / > 15/06/11 11:00:17 INFO hdfs.PeerCache: SocketCache disabled. > Found 9 items > -rw-r--r-- 3 hdfs supergroup 0 2015-06-08 12:10 > /PRE_CREATE_DIR.SUCCESS > drwxr-x--- - flume hadoop 0 2015-06-08 12:08 /flume > drwx-- - hbase hadoop 0 2015-06-10 15:25 /hbase > drwxr-xr-x - hdfs supergroup 0 2015-06-10 17:19 /hyt > drwxrwxrwx - mapred hadoop 0 2015-06-08 12:08 /mr-history > drwxr-xr-x - hdfs supergroup 0 2015-06-09 22:10 > /result_1433858936 > drwxrwxrwx - spark supergroup 0 2015-06-10 19:15 /sparkJobHistory > drwxrwxrwx - hdfs hadoop 0 2015-06-08 12:14 /tmp > drwxrwxrwx - hdfs hadoop 0 2015-06-09 21:57 /user > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count / > 15/06/11 11:00:24 INFO hdfs.PeerCache: SocketCache disabled. > 1043 171536 1756375688 / > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /PRE_CREATE_DIR.SUCCESS > 15/06/11 11:00:30 INFO hdfs.PeerCache: SocketCache disabled. >01 0 /PRE_CREATE_DIR.SUCCESS > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /flume > 15/06/11 11:00:41 INFO hdfs.PeerCache: SocketCache disabled. >10 0 /flume > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /hbase > 15/06/11 11:00:49 INFO hdfs.PeerCache: SocketCache disabled. > 36 18 14807 /hbase > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /hyt > 15/06/11 11:01:09 INFO hdfs.PeerCache: SocketCache disabled. >10 0 /hyt > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /mr-history > 15/06/11 11:01:18 INFO hdfs.PeerCache: SocketCache disabled. >30 0 /mr-history > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /result_1433858936 > 15/06/11 11:01:29 INFO hdfs.PeerCache: SocketCache disabled. > 1001 171517 1756360881 /result_1433858936 > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /sparkJobHistory > 15/06/11 11:01:41 INFO hdfs.PeerCache: SocketCache disabled. >13 21785 /sparkJobHistory > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /tmp > 15/06/11 11:01:48 INFO hdfs.PeerCache: SocketCache disabled. > 176 35958 /tmp > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /user > 15/06/11 11:01:55 INFO hdfs.PeerCache: SocketCache disabled. > 121 19077 /user -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8581) ContentSummary on / skips further counts on yielding lock
[ https://issues.apache.org/jira/browse/HDFS-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8581: -- Component/s: (was: HDFS) namenode > ContentSummary on / skips further counts on yielding lock > - > > Key: HDFS-8581 > URL: https://issues.apache.org/jira/browse/HDFS-8581 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: tongshiquan >Assignee: J.Andreina >Priority: Minor > Fix For: 2.8.0 > > Attachments: HDFS-8581.1.patch, HDFS-8581.2.patch, HDFS-8581.3.patch, > HDFS-8581.4.patch > > > If one directory such as "/result" exists about 20 files, then when > execute "hdfs dfs -count /", the result will go wrong. For all directories > whose name after "/result", file num will not be included. > My cluster see as below, "/result_1433858936" is the directory exist huge > files, and files in "/sparkJobHistory", "/tmp", "/user" are not included > vm-221:/export1/BigData/current # hdfs dfs -ls / > 15/06/11 11:00:17 INFO hdfs.PeerCache: SocketCache disabled. > Found 9 items > -rw-r--r-- 3 hdfs supergroup 0 2015-06-08 12:10 > /PRE_CREATE_DIR.SUCCESS > drwxr-x--- - flume hadoop 0 2015-06-08 12:08 /flume > drwx-- - hbase hadoop 0 2015-06-10 15:25 /hbase > drwxr-xr-x - hdfs supergroup 0 2015-06-10 17:19 /hyt > drwxrwxrwx - mapred hadoop 0 2015-06-08 12:08 /mr-history > drwxr-xr-x - hdfs supergroup 0 2015-06-09 22:10 > /result_1433858936 > drwxrwxrwx - spark supergroup 0 2015-06-10 19:15 /sparkJobHistory > drwxrwxrwx - hdfs hadoop 0 2015-06-08 12:14 /tmp > drwxrwxrwx - hdfs hadoop 0 2015-06-09 21:57 /user > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count / > 15/06/11 11:00:24 INFO hdfs.PeerCache: SocketCache disabled. > 1043 171536 1756375688 / > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /PRE_CREATE_DIR.SUCCESS > 15/06/11 11:00:30 INFO hdfs.PeerCache: SocketCache disabled. >01 0 /PRE_CREATE_DIR.SUCCESS > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /flume > 15/06/11 11:00:41 INFO hdfs.PeerCache: SocketCache disabled. >10 0 /flume > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /hbase > 15/06/11 11:00:49 INFO hdfs.PeerCache: SocketCache disabled. > 36 18 14807 /hbase > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /hyt > 15/06/11 11:01:09 INFO hdfs.PeerCache: SocketCache disabled. >10 0 /hyt > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /mr-history > 15/06/11 11:01:18 INFO hdfs.PeerCache: SocketCache disabled. >30 0 /mr-history > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /result_1433858936 > 15/06/11 11:01:29 INFO hdfs.PeerCache: SocketCache disabled. > 1001 171517 1756360881 /result_1433858936 > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /sparkJobHistory > 15/06/11 11:01:41 INFO hdfs.PeerCache: SocketCache disabled. >13 21785 /sparkJobHistory > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /tmp > 15/06/11 11:01:48 INFO hdfs.PeerCache: SocketCache disabled. > 176 35958 /tmp > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /user > 15/06/11 11:01:55 INFO hdfs.PeerCache: SocketCache disabled. > 121 19077 /user -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8581) ContentSummary on / skips further counts on yielding lock
[ https://issues.apache.org/jira/browse/HDFS-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-8581: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Committed to trunk and branch-2. thanks for the contribution [~andreina]. > ContentSummary on / skips further counts on yielding lock > - > > Key: HDFS-8581 > URL: https://issues.apache.org/jira/browse/HDFS-8581 > Project: Hadoop HDFS > Issue Type: Bug > Components: HDFS >Reporter: tongshiquan >Assignee: J.Andreina >Priority: Minor > Fix For: 2.8.0 > > Attachments: HDFS-8581.1.patch, HDFS-8581.2.patch, HDFS-8581.3.patch, > HDFS-8581.4.patch > > > If one directory such as "/result" exists about 20 files, then when > execute "hdfs dfs -count /", the result will go wrong. For all directories > whose name after "/result", file num will not be included. > My cluster see as below, "/result_1433858936" is the directory exist huge > files, and files in "/sparkJobHistory", "/tmp", "/user" are not included > vm-221:/export1/BigData/current # hdfs dfs -ls / > 15/06/11 11:00:17 INFO hdfs.PeerCache: SocketCache disabled. > Found 9 items > -rw-r--r-- 3 hdfs supergroup 0 2015-06-08 12:10 > /PRE_CREATE_DIR.SUCCESS > drwxr-x--- - flume hadoop 0 2015-06-08 12:08 /flume > drwx-- - hbase hadoop 0 2015-06-10 15:25 /hbase > drwxr-xr-x - hdfs supergroup 0 2015-06-10 17:19 /hyt > drwxrwxrwx - mapred hadoop 0 2015-06-08 12:08 /mr-history > drwxr-xr-x - hdfs supergroup 0 2015-06-09 22:10 > /result_1433858936 > drwxrwxrwx - spark supergroup 0 2015-06-10 19:15 /sparkJobHistory > drwxrwxrwx - hdfs hadoop 0 2015-06-08 12:14 /tmp > drwxrwxrwx - hdfs hadoop 0 2015-06-09 21:57 /user > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count / > 15/06/11 11:00:24 INFO hdfs.PeerCache: SocketCache disabled. > 1043 171536 1756375688 / > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /PRE_CREATE_DIR.SUCCESS > 15/06/11 11:00:30 INFO hdfs.PeerCache: SocketCache disabled. >01 0 /PRE_CREATE_DIR.SUCCESS > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /flume > 15/06/11 11:00:41 INFO hdfs.PeerCache: SocketCache disabled. >10 0 /flume > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /hbase > 15/06/11 11:00:49 INFO hdfs.PeerCache: SocketCache disabled. > 36 18 14807 /hbase > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /hyt > 15/06/11 11:01:09 INFO hdfs.PeerCache: SocketCache disabled. >10 0 /hyt > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /mr-history > 15/06/11 11:01:18 INFO hdfs.PeerCache: SocketCache disabled. >30 0 /mr-history > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /result_1433858936 > 15/06/11 11:01:29 INFO hdfs.PeerCache: SocketCache disabled. > 1001 171517 1756360881 /result_1433858936 > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /sparkJobHistory > 15/06/11 11:01:41 INFO hdfs.PeerCache: SocketCache disabled. >13 21785 /sparkJobHistory > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /tmp > 15/06/11 11:01:48 INFO hdfs.PeerCache: SocketCache disabled. > 176 35958 /tmp > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /user > 15/06/11 11:01:55 INFO hdfs.PeerCache: SocketCache disabled. > 121 19077 /user -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8581) ContentSummary on / skips further counts on yielding lock
[ https://issues.apache.org/jira/browse/HDFS-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-8581: Summary: ContentSummary on / skips further counts on yielding lock (was: count cmd calculate wrong when huge files exist in one folder) > ContentSummary on / skips further counts on yielding lock > - > > Key: HDFS-8581 > URL: https://issues.apache.org/jira/browse/HDFS-8581 > Project: Hadoop HDFS > Issue Type: Bug > Components: HDFS >Reporter: tongshiquan >Assignee: J.Andreina >Priority: Minor > Attachments: HDFS-8581.1.patch, HDFS-8581.2.patch, HDFS-8581.3.patch, > HDFS-8581.4.patch > > > If one directory such as "/result" exists about 20 files, then when > execute "hdfs dfs -count /", the result will go wrong. For all directories > whose name after "/result", file num will not be included. > My cluster see as below, "/result_1433858936" is the directory exist huge > files, and files in "/sparkJobHistory", "/tmp", "/user" are not included > vm-221:/export1/BigData/current # hdfs dfs -ls / > 15/06/11 11:00:17 INFO hdfs.PeerCache: SocketCache disabled. > Found 9 items > -rw-r--r-- 3 hdfs supergroup 0 2015-06-08 12:10 > /PRE_CREATE_DIR.SUCCESS > drwxr-x--- - flume hadoop 0 2015-06-08 12:08 /flume > drwx-- - hbase hadoop 0 2015-06-10 15:25 /hbase > drwxr-xr-x - hdfs supergroup 0 2015-06-10 17:19 /hyt > drwxrwxrwx - mapred hadoop 0 2015-06-08 12:08 /mr-history > drwxr-xr-x - hdfs supergroup 0 2015-06-09 22:10 > /result_1433858936 > drwxrwxrwx - spark supergroup 0 2015-06-10 19:15 /sparkJobHistory > drwxrwxrwx - hdfs hadoop 0 2015-06-08 12:14 /tmp > drwxrwxrwx - hdfs hadoop 0 2015-06-09 21:57 /user > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count / > 15/06/11 11:00:24 INFO hdfs.PeerCache: SocketCache disabled. > 1043 171536 1756375688 / > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /PRE_CREATE_DIR.SUCCESS > 15/06/11 11:00:30 INFO hdfs.PeerCache: SocketCache disabled. >01 0 /PRE_CREATE_DIR.SUCCESS > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /flume > 15/06/11 11:00:41 INFO hdfs.PeerCache: SocketCache disabled. >10 0 /flume > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /hbase > 15/06/11 11:00:49 INFO hdfs.PeerCache: SocketCache disabled. > 36 18 14807 /hbase > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /hyt > 15/06/11 11:01:09 INFO hdfs.PeerCache: SocketCache disabled. >10 0 /hyt > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /mr-history > 15/06/11 11:01:18 INFO hdfs.PeerCache: SocketCache disabled. >30 0 /mr-history > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /result_1433858936 > 15/06/11 11:01:29 INFO hdfs.PeerCache: SocketCache disabled. > 1001 171517 1756360881 /result_1433858936 > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /sparkJobHistory > 15/06/11 11:01:41 INFO hdfs.PeerCache: SocketCache disabled. >13 21785 /sparkJobHistory > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /tmp > 15/06/11 11:01:48 INFO hdfs.PeerCache: SocketCache disabled. > 176 35958 /tmp > vm-221:/export1/BigData/current # > vm-221:/export1/BigData/current # hdfs dfs -count /user > 15/06/11 11:01:55 INFO hdfs.PeerCache: SocketCache disabled. > 121 19077 /user -- This message was sent by Atlassian JIRA (v6.3.4#6332)
