[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339937#comment-15339937 ] Vinayakumar B commented on HDFS-9530: - Thats good to start commit with his one of the long waited

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339927#comment-15339927 ] Arpit Agarwal commented on HDFS-9530: - Sorry about that. I will hold off committing this in case

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339926#comment-15339926 ] Vinayakumar B commented on HDFS-9530: - One of the checkstyle comments can be fixed before final commit.

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339919#comment-15339919 ] Vinayakumar B commented on HDFS-9530: - Thanks for reconfirming Arpit. I too had tested it earlier

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339903#comment-15339903 ] Arpit Agarwal commented on HDFS-9530: - I take that back. I tested this again and verified the

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339872#comment-15339872 ] Arpit Agarwal commented on HDFS-9530: - Hi [~brahmareddy], the test case is still not exercising the

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339624#comment-15339624 ] Brahma Reddy Battula commented on HDFS-9530: Thanks [~vinayrpet] for your review. I will wait

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339360#comment-15339360 ] Hadoop QA commented on HDFS-9530: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem ||

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339339#comment-15339339 ] Vinayakumar B commented on HDFS-9530: - Latest patch looks good. +1. Checkstyle and whitespace comments

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339249#comment-15339249 ] Brahma Reddy Battula commented on HDFS-9530: {{Test failures}},{{checkstyle}} and {{ASF License

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336382#comment-15336382 ] Hadoop QA commented on HDFS-9530: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem ||

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336125#comment-15336125 ] Brahma Reddy Battula commented on HDFS-9530: [~arpitagarwal] thanks for catch. bq.The

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332219#comment-15332219 ] Arpit Agarwal commented on HDFS-9530: - Thanks [~brahmareddy], the core change looks fine to me. The

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331251#comment-15331251 ] Brahma Reddy Battula commented on HDFS-9530: Thanks to all.Uploaded patch.. Attached test case

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328189#comment-15328189 ] Ravi Prakash commented on HDFS-9530: This has been a long standing and complicated problem and your

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324305#comment-15324305 ] Vinayakumar B commented on HDFS-9530: - bq. Also agree with Vinayakumar B that releasing via invalidate

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324304#comment-15324304 ] Vinayakumar B commented on HDFS-9530: - bq. Do we agree on the following summary of the contract for

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321609#comment-15321609 ] Arpit Agarwal commented on HDFS-9530: - I took a deeper look at the reservation code. It was painful to

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299597#comment-15299597 ] Vinayakumar B commented on HDFS-9530: - Good analysis [~brahmareddy]. Your analysis makes sense. IMO

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249142#comment-15249142 ] Ravi Prakash commented on HDFS-9530: Perhaps we can postpone the question of whether RBW blocks which

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15248968#comment-15248968 ] Arpit Agarwal commented on HDFS-9530: - Thanks for continuing to work on this [~brahmareddy] and your

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247619#comment-15247619 ] Brahma Reddy Battula commented on HDFS-9530: bq. Do you think we should change

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15246697#comment-15246697 ] Ravi Prakash commented on HDFS-9530: To answer one of my own questions: "Could you please point me to

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15246179#comment-15246179 ] Ravi Prakash commented on HDFS-9530: bq. 1. Reservation happens only when the block is being received

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239582#comment-15239582 ] Brahma Reddy Battula commented on HDFS-9530: anybody can review the attached patch..? > huge

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15231505#comment-15231505 ] Brahma Reddy Battula commented on HDFS-9530: To Make it simple for analysis, 1. Reservation

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15231242#comment-15231242 ] Ravi Prakash commented on HDFS-9530: Thanks for your attention and patch Brahma! On a brief glance,

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15231164#comment-15231164 ] Ravi Prakash commented on HDFS-9530: FWIW, there is only a trivial amount of data in the RBW directory

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15229542#comment-15229542 ] Brahma Reddy Battula commented on HDFS-9530: If any of the datanode other than 1st DN is down

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15229271#comment-15229271 ] Arpit Agarwal commented on HDFS-9530: - Hi [~brahmareddy], I didn't quite understand the patch. Could

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15228595#comment-15228595 ] Ravi Prakash commented on HDFS-9530: Linking these two JIRAs together. Maybe they are related? > huge

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15228360#comment-15228360 ] Arpit Agarwal commented on HDFS-9530: - Hi [~brahmareddy], yes I meant HDFS-8626 thanks for finding it.

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1522#comment-1522 ] Brahma Reddy Battula commented on HDFS-9530: [~arpitagarwal] are you referring HDFS-8626..?

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227467#comment-15227467 ] Arpit Agarwal commented on HDFS-9530: - I think there was a related fix to HDFS-8072. I can't recall the

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227441#comment-15227441 ] Ravi Prakash commented on HDFS-9530: HDFS-8072 has not fixed this for us. [~arpitagarwal] know anything

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227402#comment-15227402 ] Ravi Prakash commented on HDFS-9530: I took a heap dump of the Datanode process. I see the value of

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227081#comment-15227081 ] Ravi Prakash commented on HDFS-9530: Thanks for reporting Fei Hui. I can confirm that we are seeing

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186922#comment-15186922 ] Fei Hui commented on HDFS-9530: --- while job running, the df output is above.there is no non-dfs used, that is

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186846#comment-15186846 ] Brahma Reddy Battula commented on HDFS-9530: is this will be there after hive job is

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186836#comment-15186836 ] Fei Hui commented on HDFS-9530: --- execute df & dfs report in order after the job finished， non dfs usage is

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186819#comment-15186819 ] Akira AJISAKA commented on HDFS-9530: - Hi [~ferhui], did you execute df command when running the job? I

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186764#comment-15186764 ] Fei Hui commented on HDFS-9530: --- maybe it's different from HDFS-9038. the example in description,

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186626#comment-15186626 ] Akira AJISAKA commented on HDFS-9530: - bq. Should we resolve this as a duplicate of HDFS-9038? Agree.

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154036#comment-15154036 ] Qiuzhuang Lian commented on HDFS-9530: -- @Brahma Reddy Battula, thanks. For now we restart all

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153893#comment-15153893 ] Brahma Reddy Battula commented on HDFS-9530: [~Qiuzhuang] this issue is tracked in HDFS-9038..

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153875#comment-15153875 ] Qiuzhuang Lian commented on HDFS-9530: -- we see this problem too in hadoop 2.6.2. The dfs non used is

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059761#comment-15059761 ] Fei Hui commented on HDFS-9530: --- freeSpace conflicts with df putputs where is freeSpace code? maybe there is

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059760#comment-15059760 ] Fei Hui commented on HDFS-9530: --- freeSpace conflicts with df putputs where is freeSpace code? maybe there is

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050286#comment-15050286 ] Fei Hui commented on HDFS-9530: --- [hdfs@worker-1 ~]$ hadoop version Hadoop 2.6.2 and i am sure it also

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050453#comment-15050453 ] Fei Hui commented on HDFS-9530: --- dfs.namenode.resource.du.reserved 1073741824 > huge Non-DFS

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15052235#comment-15052235 ] Fei Hui commented on HDFS-9530: --- useful information， jmx result conflicts with df "VolumeInfo" :

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048384#comment-15048384 ] Fei Hui commented on HDFS-9530: --- i want to add logs to monitor DataNode DFS Used, Non DFS Used,and so on. how

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048653#comment-15048653 ] Brahma Reddy Battula commented on HDFS-9530: [~ferhui] thanks for reporting this. Can you

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049367#comment-15049367 ] Chris Nauroth commented on HDFS-9530: - Additionally, Apache Hadoop 2.7.1 has a bug that configured

[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1

[ https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049010#comment-15049010 ] Xiaoyu Yao commented on HDFS-9530: -- This looks like the symptom of HDFS-8072, where RBW reserved space is