[jira] [Commented] (HDFS-15735) NameNode memory Leak on frequent execution of fsck
[ https://issues.apache.org/jira/browse/HDFS-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313300#comment-17313300 ] Brahma Reddy Battula commented on HDFS-15735: - if there are no objects, can I commit this one on this week..? > NameNode memory Leak on frequent execution of fsck > > > Key: HDFS-15735 > URL: https://issues.apache.org/jira/browse/HDFS-15735 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ravuri Sushma sree >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HDFS-15735.001.patch > > > The memory of the cluster NameNode continues to grow, and the full gc > eventually leads to the failure of the active and standby HDFS > Htrace is used to track the processing time of fsck > Checking the code it is found that the tracer object in NamenodeFsck.java was > only created but not closed because of this the memory footprint continues to > grow -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15735) NameNode memory Leak on frequent execution of fsck
[
https://issues.apache.org/jira/browse/HDFS-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283235#comment-17283235
]
Brahma Reddy Battula commented on HDFS-15735:
-
{quote} Tracer is a {{private}} variable,
Not used anywhere,
Tracer is subject to removal due to CVE(IIRC), guess HADOOP-17387 and others,
one recently mentioned too.
{quote}
Looks you did not get what does we mean, we talked about this config
*"namenode.fsck.htrace."*
{quote}Harmless things are not always correct,
closing tracer in fsck() may impact if someone is using tracer post it(if so).
Closing in the last line of fsck may not be this issue what you are fixing. the
moment you come out from the method control, the tracer would be subject to GC?
closing it won't help, it will also make it subject to GC only.
{quote}
How it will impact for this ..?
{quote}Would request consider the other options as well.
{quote}
Let's anybody else have objection to go this.
{quote}On this note, I take my vote back.
{quote}
thanks.
> NameNode memory Leak on frequent execution of fsck
>
>
> Key: HDFS-15735
> URL: https://issues.apache.org/jira/browse/HDFS-15735
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ravuri Sushma sree
>Assignee: Ravuri Sushma sree
>Priority: Major
> Attachments: HDFS-15735.001.patch
>
>
> The memory of the cluster NameNode continues to grow, and the full gc
> eventually leads to the failure of the active and standby HDFS
> Htrace is used to track the processing time of fsck
> Checking the code it is found that the tracer object in NamenodeFsck.java was
> only created but not closed because of this the memory footprint continues to
> grow
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15735) NameNode memory Leak on frequent execution of fsck
[
https://issues.apache.org/jira/browse/HDFS-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258131#comment-17258131
]
Ayush Saxena commented on HDFS-15735:
-
Tracer is a {{private}} variable,
Not used anywhere,
Tracer is subject to removal due to CVE(IIRC), guess HADOOP-17387 and others,
one recently mentioned too.
Harmless things are not always correct,
closing tracer in fsck() may impact if someone is using tracer post it(if so).
Closing in the last line of fsck may not be this issue what you are fixing. the
moment you come out from the method control, the tracer would be subject to GC?
closing it won't help, it will also make it subject to GC only.
If someone is using tracer in internal code, not in OS, or there is no where to
use here, no need to keep, the guy can keep in his internal code.
Removal will save memory allocation, and isn't incompatible in any way. Would
be even better.
Sometimes listening to others doesn't hurt, not me atleast [~John Smith] too
had a comment.
** Now the catch, I will still respect your opinion on this, though you aren't
interested in mine :( You won't see a committing shortly from my end unless you
are convinced in "any" jira. and I don't claim I am correct here, just
proposing something if it looks good, can be done. I can be *wrong*/completely
wrong, would be happy to accept that.
Would request consider the other options as well. I shall be happy to connect
with you offline as well, If you want.
On this note, I take my vote back.
> NameNode memory Leak on frequent execution of fsck
>
>
> Key: HDFS-15735
> URL: https://issues.apache.org/jira/browse/HDFS-15735
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ravuri Sushma sree
>Assignee: Ravuri Sushma sree
>Priority: Major
> Attachments: HDFS-15735.001.patch
>
>
> The memory of the cluster NameNode continues to grow, and the full gc
> eventually leads to the failure of the active and standby HDFS
> Htrace is used to track the processing time of fsck
> Checking the code it is found that the tracer object in NamenodeFsck.java was
> only created but not closed because of this the memory footprint continues to
> grow
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15735) NameNode memory Leak on frequent execution of fsck
[
https://issues.apache.org/jira/browse/HDFS-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258055#comment-17258055
]
Brahma Reddy Battula commented on HDFS-15735:
-
{quote}I am not sure about it, why it being configurable makes it necessary to
be here, why closing is better. Please hold it.
-1
{quote}
Removal can impact existing user whoever use this feature as they've configured
and Proposed fix will not break anything.
Not sure why this needs to hold and given -1 and I feel this will not good
practice.
> NameNode memory Leak on frequent execution of fsck
>
>
> Key: HDFS-15735
> URL: https://issues.apache.org/jira/browse/HDFS-15735
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ravuri Sushma sree
>Assignee: Ravuri Sushma sree
>Priority: Major
> Attachments: HDFS-15735.001.patch
>
>
> The memory of the cluster NameNode continues to grow, and the full gc
> eventually leads to the failure of the active and standby HDFS
> Htrace is used to track the processing time of fsck
> Checking the code it is found that the tracer object in NamenodeFsck.java was
> only created but not closed because of this the memory footprint continues to
> grow
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15735) NameNode memory Leak on frequent execution of fsck
[ https://issues.apache.org/jira/browse/HDFS-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253448#comment-17253448 ] Ayush Saxena commented on HDFS-15735: - I am not sure about it, why it being configurable makes it necessary to be here, why closing is better. Please hold it. -1 > NameNode memory Leak on frequent execution of fsck > > > Key: HDFS-15735 > URL: https://issues.apache.org/jira/browse/HDFS-15735 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ravuri Sushma sree >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HDFS-15735.001.patch > > > The memory of the cluster NameNode continues to grow, and the full gc > eventually leads to the failure of the active and standby HDFS > Htrace is used to track the processing time of fsck > Checking the code it is found that the tracer object in NamenodeFsck.java was > only created but not closed because of this the memory footprint continues to > grow -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15735) NameNode memory Leak on frequent execution of fsck
[ https://issues.apache.org/jira/browse/HDFS-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253447#comment-17253447 ] Hemanth Boyina commented on HDFS-15735: --- +1 will commit shortly > NameNode memory Leak on frequent execution of fsck > > > Key: HDFS-15735 > URL: https://issues.apache.org/jira/browse/HDFS-15735 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ravuri Sushma sree >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HDFS-15735.001.patch > > > The memory of the cluster NameNode continues to grow, and the full gc > eventually leads to the failure of the active and standby HDFS > Htrace is used to track the processing time of fsck > Checking the code it is found that the tracer object in NamenodeFsck.java was > only created but not closed because of this the memory footprint continues to > grow -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15735) NameNode memory Leak on frequent execution of fsck
[ https://issues.apache.org/jira/browse/HDFS-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252618#comment-17252618 ] Brahma Reddy Battula commented on HDFS-15735: - [~Sushma_28] thanks for reporting. Closing tracer is better as of now. LGTM. > NameNode memory Leak on frequent execution of fsck > > > Key: HDFS-15735 > URL: https://issues.apache.org/jira/browse/HDFS-15735 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ravuri Sushma sree >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HDFS-15735.001.patch > > > The memory of the cluster NameNode continues to grow, and the full gc > eventually leads to the failure of the active and standby HDFS > Htrace is used to track the processing time of fsck > Checking the code it is found that the tracer object in NamenodeFsck.java was > only created but not closed because of this the memory footprint continues to > grow -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15735) NameNode memory Leak on frequent execution of fsck
[ https://issues.apache.org/jira/browse/HDFS-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251762#comment-17251762 ] Ravuri Sushma sree commented on HDFS-15735: --- [~ayushtkn] [~John Smith] , Thank you for the reviews. I think here tracing is configurable so I opted to close it instead of removing and stopping the call of fsck may vary depending on different business requirements. > NameNode memory Leak on frequent execution of fsck > > > Key: HDFS-15735 > URL: https://issues.apache.org/jira/browse/HDFS-15735 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ravuri Sushma sree >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HDFS-15735.001.patch > > > The memory of the cluster NameNode continues to grow, and the full gc > eventually leads to the failure of the active and standby HDFS > Htrace is used to track the processing time of fsck > Checking the code it is found that the tracer object in NamenodeFsck.java was > only created but not closed because of this the memory footprint continues to > grow -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15735) NameNode memory Leak on frequent execution of fsck
[ https://issues.apache.org/jira/browse/HDFS-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251445#comment-17251445 ] Yuxuan Wang commented on HDFS-15735: [~ayushtkn] I don't find any usage of tracer. Removing it is more sensible to me. > NameNode memory Leak on frequent execution of fsck > > > Key: HDFS-15735 > URL: https://issues.apache.org/jira/browse/HDFS-15735 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ravuri Sushma sree >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HDFS-15735.001.patch > > > The memory of the cluster NameNode continues to grow, and the full gc > eventually leads to the failure of the active and standby HDFS > Htrace is used to track the processing time of fsck > Checking the code it is found that the tracer object in NamenodeFsck.java was > only created but not closed because of this the memory footprint continues to > grow -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15735) NameNode memory Leak on frequent execution of fsck
[
https://issues.apache.org/jira/browse/HDFS-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251313#comment-17251313
]
Hadoop QA commented on HDFS-15735:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
48s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
8s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 18s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
8s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config;
considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
6s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 46s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
|
[jira] [Commented] (HDFS-15735) NameNode memory Leak on frequent execution of fsck
[
https://issues.apache.org/jira/browse/HDFS-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251243#comment-17251243
]
Ayush Saxena commented on HDFS-15735:
-
Is that Tracer used somewhere?
I guess HDFS-13702, removed the only usage? If it isn't used anywhere we can
remove the variable?
If it is used, Then is closing in {{fsck()}} a correct choice? Can I not call
{{fsck()}} more than once, or anything that uses tracer, post calling fsck.
> NameNode memory Leak on frequent execution of fsck
>
>
> Key: HDFS-15735
> URL: https://issues.apache.org/jira/browse/HDFS-15735
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ravuri Sushma sree
>Assignee: Ravuri Sushma sree
>Priority: Major
> Attachments: HDFS-15735.001.patch
>
>
> The memory of the cluster NameNode continues to grow, and the full gc
> eventually leads to the failure of the active and standby HDFS
> Htrace is used to track the processing time of fsck
> Checking the code it is found that the tracer object in NamenodeFsck.java was
> only created but not closed because of this the memory footprint continues to
> grow
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15735) NameNode memory Leak on frequent execution of fsck
[ https://issues.apache.org/jira/browse/HDFS-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251200#comment-17251200 ] Ravuri Sushma sree commented on HDFS-15735: --- Attached a patch closing tracer object in fsck(). Please review > NameNode memory Leak on frequent execution of fsck > > > Key: HDFS-15735 > URL: https://issues.apache.org/jira/browse/HDFS-15735 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ravuri Sushma sree >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HDFS-15735.001.patch > > > The memory of the cluster NameNode continues to grow, and the full gc > eventually leads to the failure of the active and standby HDFS > Htrace is used to track the processing time of fsck > Checking the code it is found that the tracer object in NamenodeFsck.java was > only created but not closed because of this the memory footprint continues to > grow -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
