[
https://issues.apache.org/jira/browse/TEZ-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062996#comment-17062996
]
TezQA commented on TEZ-4087:
----------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
0s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange} tez-runtime-library: The patch generated 2 new +
99 unchanged - 0 fixed = 101 total (was 99) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
58s{color} | {color:red} tez-runtime-library generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
28s{color} | {color:green} tez-runtime-library in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:tez-runtime-library |
| | Inconsistent synchronization of
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.shuffleSchedulerThread;
locked 80% of time Unsynchronized access at MergeThread.java:80% of time
Unsynchronized access at MergeThread.java:[line 66] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 base:
https://builds.apache.org/job/PreCommit-TEZ-Build/319/artifact/out/Dockerfile |
| JIRA Issue | TEZ-4087 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12997171/TEZ-4087.1.patch |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs
checkstyle compile |
| uname | Linux 6fc9bccb9fc2 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / b5b432b |
| Default Java | 1.8.0_242 |
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/319/artifact/out/diff-checkstyle-tez-runtime-library.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-TEZ-Build/319/artifact/out/new-findbugs-tez-runtime-library.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-TEZ-Build/319/testReport/ |
| Max. process+thread count | 126 (vs. ulimit of 5500) |
| modules | C: tez-runtime-library U: tez-runtime-library |
| Console output |
https://builds.apache.org/job/PreCommit-TEZ-Build/319/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> Shuffle: Fix shuffle cleanup to prevent thread leaks
> ----------------------------------------------------
>
> Key: TEZ-4087
> URL: https://issues.apache.org/jira/browse/TEZ-4087
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Major
> Attachments: TEZ-4087.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In certain cases, Shuffle's cleanupIgnoreErrors() is not called. This leaves
> 4 threads (inmem, diskmerger, Referee, ShuffleAndMergeRunner) run forever.
> When these are run in long running processes (e.g LLAP in Hive), they reach
> the thread limits over time.
> Note: Root cause why cleanupIgnoreErrors() is not invoked is not yet known. I
> will share the details when i get more details on this. Creating this ticket
> to add additional safety knobs to ensure that thread leaks do not happen.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
