[jira] [Commented] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361799#comment-17361799
]
Jim Brennan commented on HADOOP-17758:
--
I don't think the NPE will happen with HADOOP-17728 reverted. I think we can
close this as fixed by reverting HADOOP-17728.
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361537#comment-17361537
]
Hadoop QA commented on HADOOP-17758:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue} 0m
0s{color} | | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | | {color:red} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 33m
25s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
39s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
16s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m
24s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 2s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
58s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 21m
58s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
19s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m
19s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m
32s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 17s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
50s{color} | | {color:green} hadoop-common in the patch passed.
[jira] [Commented] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361509#comment-17361509
]
Mingliang Liu commented on HADOOP-17758:
I have not checked the failure detail here carefully, but clearly the NPE is a
bug. I have reverted the original code in 3.2/3.3/3.4 branches. Thanks for
reporting.
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361364#comment-17361364
]
yikf commented on HADOOP-17758:
---
Oh, Thanks, you are right, My bad, If something is added to the queue,
ReferenceQueue.enqueue will be called, i.e. lock.notifyAll be called, That will
not wait forerver.
I will revert.
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361075#comment-17361075
]
Jim Brennan commented on HADOOP-17758:
--
{quote}
We need a timeout to wake me up.
{quote}
I still don't understand why you need to wake up if the queue is still empty.
There is nothing to clean up until something is added to the queue.
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361068#comment-17361068
]
yikf commented on HADOOP-17758:
---
Cleanup is a persistent process. If the queue is empty, the current thread will
wait forever It's not a deadlock, though and not notify, so cleanUp will not
work.
We need a timeout to wake me up.
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361053#comment-17361053
]
Jim Brennan commented on HADOOP-17758:
--
I'm sorry, but am I missing something? What is the point of using a timeout,
if nothing has been added to the queue? We'll just loop around again and wait
on the same lock until something does get queued.
One thread waiting on a lock is not a deadlock. Is this causing some other
thread to block as well?
My inclination would be to revert [HADOOP-17728].
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361023#comment-17361023
]
yikf commented on HADOOP-17758:
---
Thanks for your reporting, We change it beacuse that It will wait forerver if
we have't the timeout and the RerferenceQueue is empty;
eg (Suppose we hav't the timeout):
# we call the RerferenceQueue.remove()
# if we poll is empty from the queue, RerferenceQueue will
lock.wait(0)
# cleanUp cant work due to the DeadLock
>From above, we nee the timeout to prevent deadLock.
reference JDK ReferenctQueue:
{code:java}
public Reference remove(long timeout)
throws IllegalArgumentException, InterruptedException
{
if (timeout < 0) {
throw new IllegalArgumentException("Negative timeout value");
}
synchronized (lock) {
Reference r = reallyPoll();
if (r != null) return r;
long start = (timeout == 0) ? 0 : System.nanoTime();
for (;;) {
lock.wait(timeout);
r = reallyPoll();
if (r != null) return r;
if (timeout != 0) {
long end = System.nanoTime();
timeout -= (end - start) / 1000_000;
if (timeout <= 0) return null;
start = end;
}
}
}
}
{code}
But i ignore it that the return of method of the remove may be is null, i will
fix the issue, Thanks~
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
