[jira] [Updated] (TEZ-4204) Data race in RootInputInitializerManager
[ https://issues.apache.org/jira/browse/TEZ-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated TEZ-4204: -- Fix Version/s: 0.10.1 > Data race in RootInputInitializerManager > > > Key: TEZ-4204 > URL: https://issues.apache.org/jira/browse/TEZ-4204 > Project: Apache Tez > Issue Type: Bug >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Blocker > Fix For: 0.10.1 > > Attachments: TEZ-4204.1.patch, TEZ-4204.1.patch, TEZ-4204.2.patch > > > After https://issues.apache.org/jira/browse/TEZ-4170 there is a data race for > initializerMap in RootInputInitializerManager. initializerMap should be > initialized before vertex state is set to initializing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4204) Data race in RootInputInitializerManager
[ https://issues.apache.org/jira/browse/TEZ-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168413#comment-17168413 ] László Bodor commented on TEZ-4204: --- ok, resolving this, thanks again for the patch and [~ashutoshc] for the review! > Data race in RootInputInitializerManager > > > Key: TEZ-4204 > URL: https://issues.apache.org/jira/browse/TEZ-4204 > Project: Apache Tez > Issue Type: Bug >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Blocker > Attachments: TEZ-4204.1.patch, TEZ-4204.1.patch, TEZ-4204.2.patch > > > After https://issues.apache.org/jira/browse/TEZ-4170 there is a data race for > initializerMap in RootInputInitializerManager. initializerMap should be > initialized before vertex state is set to initializing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4212) Fix build checkstyle configuration and suppressions dtd URLs
[
https://issues.apache.org/jira/browse/TEZ-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168295#comment-17168295
]
Jonathan Turner Eagles commented on TEZ-4212:
-
[~abstractdog], can you take a look at this checkstyle change? This will help
get the nightly builds working again (TEZ-4184)
> Fix build checkstyle configuration and suppressions dtd URLs
>
>
> Key: TEZ-4212
> URL: https://issues.apache.org/jira/browse/TEZ-4212
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Turner Eagles
>Assignee: Jonathan Turner Eagles
>Priority: Major
> Attachments: TEZ-4212.001.patch
>
>
> {code}
> tez-build-tools/src/main/resources/checkstyle/checkstyle.xml:
> java.lang.RuntimeException: java.io.FileNotFoundException:
> http://www.puppycrawl.com/dtds/configuration_1_2.dtd
> at
> jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:397)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:449)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:406)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:402)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.eval(NashornScriptEngine.java:155)
> at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
> at com.sun.tools.script.shell.Main.evaluateString(Main.java:298)
> at com.sun.tools.script.shell.Main.evaluateString(Main.java:319)
> at com.sun.tools.script.shell.Main.access$300(Main.java:37)
> at com.sun.tools.script.shell.Main$3.run(Main.java:217)
> at com.sun.tools.script.shell.Main.main(Main.java:48)
> Caused by: java.io.FileNotFoundException:
> http://www.puppycrawl.com/dtds/configuration_1_2.dtd
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:647)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startEntity(XMLEntityManager.java:1304)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startDTDEntity(XMLEntityManager.java:1270)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.setInputSource(XMLDTDScannerImpl.java:264)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.dispatch(XMLDocumentScannerImpl.java:1161)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.next(XMLDocumentScannerImpl.java:1045)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:959)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
> at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at
> com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
> at
> com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
> at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:205)
> at
> jdk.nashorn.internal.scripts.Script$Recompilation$2$19313A$\^system_init\_.XMLDocument(:747)
> at jdk.nashorn.internal.scripts.Script$1$\^string\_.:program(:1)
> at
> jdk.nashorn.internal.runtime.ScriptFunctionData.invoke(ScriptFunctionData.java:637)
> at
> jdk.nashorn.internal.runtime.ScriptFunction.invoke(ScriptFunction.java:494)
> at
> jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:393)
> ... 10 more
> tez-build-tools/src/main/resources/checkstyle/suppressions.xml:
> java.lang.RuntimeException: java.io.FileNotFoundException:
> http://www.puppycrawl.com/dtds/suppressions_1_1.dtd
> at
> jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:397)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:449)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:406)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:402)
> at
>
[jira] [Updated] (TEZ-4213) Bound appContext executor capacity using a configurable property
[
https://issues.apache.org/jira/browse/TEZ-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Panagiotis Garefalakis updated TEZ-4213:
Description:
After TEZ-4170 was merged, appContext executor pool is also used by the
RootInputInitializerManager to speed up SplitGeneration.
However, this executor pool currently has not capacity limit
https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java#L624
The problem the occurs when generating splits for larger inputs (thousands or
more) is that it can could result to
{color:red}java.lang.OutOfMemoryError: unable to create new native thread{color}
that is also reproducible with a test case.
https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/dag/RootInputInitializerManager.java#L130
To avoid such errors, I propose to limit the capacity of this pool to a
configurable value that can be for example the number of physical cores by
default.
was:
After TEZ-4170 was merged, appContext executor pool is also used by the
RootInputInitializerManager to speed up SplitGeneration.
However, this executor pool currently has not capacity limit
https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java#L624
The problem the occurs when generating splits for larger inputs (thousands or
more) is that it can could result to
{color:red}java.lang.OutOfMemoryError: unable to create new native thread{color}
that is also reproducible with a test case.
To avoid such errors, I propose to limit the capacity of this pool to a
configurable value that can be for example the number of physical cores by
default.
> Bound appContext executor capacity using a configurable property
>
>
> Key: TEZ-4213
> URL: https://issues.apache.org/jira/browse/TEZ-4213
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Major
>
> After TEZ-4170 was merged, appContext executor pool is also used by the
> RootInputInitializerManager to speed up SplitGeneration.
> However, this executor pool currently has not capacity limit
> https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java#L624
> The problem the occurs when generating splits for larger inputs (thousands or
> more) is that it can could result to
> {color:red}java.lang.OutOfMemoryError: unable to create new native
> thread{color}
> that is also reproducible with a test case.
> https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/dag/RootInputInitializerManager.java#L130
> To avoid such errors, I propose to limit the capacity of this pool to a
> configurable value that can be for example the number of physical cores by
> default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (TEZ-4213) Bound appContext executor capacity using a configurable property
[
https://issues.apache.org/jira/browse/TEZ-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Panagiotis Garefalakis updated TEZ-4213:
Description:
After TEZ-4170 was merged, appContext executor pool is also used by the
RootInputInitializerManager to speed up SplitGeneration.
However, this executor pool currently has not capacity limit
https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java#L624
The problem the occurs when generating splits for larger inputs (thousands or
more) is that it can could result to
{color:red}java.lang.OutOfMemoryError: unable to create new native thread{color}
that is also reproducible with a test case.
To avoid such errors, I propose to limit the capacity of this pool to a
configurable value that can be for example the number of physical cores by
default.
was:
After TEZ-4170 was merged, appContext executor pool is also used by the
RootInputInitializerManager to speed up SplitGeneration.
However, this executor pool currently has not capacity limit
https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java#L624
When generating splits for larger inputs (thousands or more) this could result
in
{color:red}java.lang.OutOfMemoryError: unable to create new native thread{color}
(reproducible with a test case)
I propose to limit the capacity of this pool to a configurable value that can
be for example the number of physical cores by default.
> Bound appContext executor capacity using a configurable property
>
>
> Key: TEZ-4213
> URL: https://issues.apache.org/jira/browse/TEZ-4213
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Major
>
> After TEZ-4170 was merged, appContext executor pool is also used by the
> RootInputInitializerManager to speed up SplitGeneration.
> However, this executor pool currently has not capacity limit
> https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java#L624
> The problem the occurs when generating splits for larger inputs (thousands or
> more) is that it can could result to
> {color:red}java.lang.OutOfMemoryError: unable to create new native
> thread{color}
> that is also reproducible with a test case.
> To avoid such errors, I propose to limit the capacity of this pool to a
> configurable value that can be for example the number of physical cores by
> default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (TEZ-4213) Bound appContext executor capacity using a configurable property
Panagiotis Garefalakis created TEZ-4213:
---
Summary: Bound appContext executor capacity using a configurable
property
Key: TEZ-4213
URL: https://issues.apache.org/jira/browse/TEZ-4213
Project: Apache Tez
Issue Type: Bug
Reporter: Panagiotis Garefalakis
Assignee: Panagiotis Garefalakis
After TEZ-4170 was merged, appContext executor pool is also used by the
RootInputInitializerManager to speed up SplitGeneration.
However, this executor pool currently has not capacity limit
https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java#L624
When generating splits for larger inputs (thousands or more) this could result
in
{color:red}java.lang.OutOfMemoryError: unable to create new native thread{color}
(reproducible with a test case)
I propose to limit the capacity of this pool to a configurable value that can
be for example the number of physical cores by default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4212) Fix build checkstyle configuration and suppressions dtd URLs
[
https://issues.apache.org/jira/browse/TEZ-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168242#comment-17168242
]
TezQA commented on TEZ-4212:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} tez-build-tools in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 59m
28s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-TEZ-Build/505/artifact/out/Dockerfile |
| JIRA Issue | TEZ-4212 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13008793/TEZ-4212.001.patch |
| Optional Tests | dupname asflicense javac javadoc unit xml compile |
| uname | Linux 39d121264253 4.15.0-101-generic #102-Ubuntu SMP Mon May 11
10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 9d2b61b57 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Test Results |
https://builds.apache.org/job/PreCommit-TEZ-Build/505/testReport/ |
| Max. process+thread count | 1517 (vs. ulimit of 5500) |
| modules | C: tez-build-tools . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-TEZ-Build/505/console |
| versions | git=2.17.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Fix build checkstyle configuration and suppressions dtd URLs
>
>
> Key: TEZ-4212
> URL: https://issues.apache.org/jira/browse/TEZ-4212
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Turner Eagles
>Assignee: Jonathan Turner Eagles
>Priority: Major
> Attachments: TEZ-4212.001.patch
>
>
> {code}
> tez-build-tools/src/main/resources/checkstyle/checkstyle.xml:
> java.lang.RuntimeException: java.io.FileNotFoundException:
[jira] [Commented] (TEZ-4212) Fix build checkstyle configuration and suppressions dtd URLs
[
https://issues.apache.org/jira/browse/TEZ-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168193#comment-17168193
]
Jonathan Turner Eagles commented on TEZ-4212:
-
This is both an upgrade to latest checkstyle and updating the url to point to
the new server location. Hopefully, the analysis in the related JIRAs will help
understand this fully. Basically, Old dtd locations were moved to a new
location. In order to move to the new location, 1) must update puppycrawl
version 2) which leads to must updating checkstyle version, 3) which leads to
updating the dtd versions, 4) which leads to modifying LineLength check to no
longer be a child of the TreeWalker parent.
> Fix build checkstyle configuration and suppressions dtd URLs
>
>
> Key: TEZ-4212
> URL: https://issues.apache.org/jira/browse/TEZ-4212
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Turner Eagles
>Assignee: Jonathan Turner Eagles
>Priority: Major
> Attachments: TEZ-4212.001.patch
>
>
> {code}
> tez-build-tools/src/main/resources/checkstyle/checkstyle.xml:
> java.lang.RuntimeException: java.io.FileNotFoundException:
> http://www.puppycrawl.com/dtds/configuration_1_2.dtd
> at
> jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:397)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:449)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:406)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:402)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.eval(NashornScriptEngine.java:155)
> at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
> at com.sun.tools.script.shell.Main.evaluateString(Main.java:298)
> at com.sun.tools.script.shell.Main.evaluateString(Main.java:319)
> at com.sun.tools.script.shell.Main.access$300(Main.java:37)
> at com.sun.tools.script.shell.Main$3.run(Main.java:217)
> at com.sun.tools.script.shell.Main.main(Main.java:48)
> Caused by: java.io.FileNotFoundException:
> http://www.puppycrawl.com/dtds/configuration_1_2.dtd
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:647)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startEntity(XMLEntityManager.java:1304)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startDTDEntity(XMLEntityManager.java:1270)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.setInputSource(XMLDTDScannerImpl.java:264)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.dispatch(XMLDocumentScannerImpl.java:1161)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.next(XMLDocumentScannerImpl.java:1045)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:959)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
> at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at
> com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
> at
> com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
> at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:205)
> at
> jdk.nashorn.internal.scripts.Script$Recompilation$2$19313A$\^system_init\_.XMLDocument(:747)
> at jdk.nashorn.internal.scripts.Script$1$\^string\_.:program(:1)
> at
> jdk.nashorn.internal.runtime.ScriptFunctionData.invoke(ScriptFunctionData.java:637)
> at
> jdk.nashorn.internal.runtime.ScriptFunction.invoke(ScriptFunction.java:494)
> at
> jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:393)
> ... 10 more
> tez-build-tools/src/main/resources/checkstyle/suppressions.xml:
> java.lang.RuntimeException: java.io.FileNotFoundException:
> http://www.puppycrawl.com/dtds/suppressions_1_1.dtd
> at
>
[jira] [Updated] (TEZ-4212) Fix build checkstyle configuration and suppressions dtd URLs
[
https://issues.apache.org/jira/browse/TEZ-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles updated TEZ-4212:
Attachment: TEZ-4212.001.patch
> Fix build checkstyle configuration and suppressions dtd URLs
>
>
> Key: TEZ-4212
> URL: https://issues.apache.org/jira/browse/TEZ-4212
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Turner Eagles
>Assignee: Jonathan Turner Eagles
>Priority: Major
> Attachments: TEZ-4212.001.patch
>
>
> {code}
> tez-build-tools/src/main/resources/checkstyle/checkstyle.xml:
> java.lang.RuntimeException: java.io.FileNotFoundException:
> http://www.puppycrawl.com/dtds/configuration_1_2.dtd
> at
> jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:397)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:449)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:406)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:402)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.eval(NashornScriptEngine.java:155)
> at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
> at com.sun.tools.script.shell.Main.evaluateString(Main.java:298)
> at com.sun.tools.script.shell.Main.evaluateString(Main.java:319)
> at com.sun.tools.script.shell.Main.access$300(Main.java:37)
> at com.sun.tools.script.shell.Main$3.run(Main.java:217)
> at com.sun.tools.script.shell.Main.main(Main.java:48)
> Caused by: java.io.FileNotFoundException:
> http://www.puppycrawl.com/dtds/configuration_1_2.dtd
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:647)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startEntity(XMLEntityManager.java:1304)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startDTDEntity(XMLEntityManager.java:1270)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.setInputSource(XMLDTDScannerImpl.java:264)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.dispatch(XMLDocumentScannerImpl.java:1161)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.next(XMLDocumentScannerImpl.java:1045)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:959)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
> at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at
> com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
> at
> com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
> at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:205)
> at
> jdk.nashorn.internal.scripts.Script$Recompilation$2$19313A$\^system_init\_.XMLDocument(:747)
> at jdk.nashorn.internal.scripts.Script$1$\^string\_.:program(:1)
> at
> jdk.nashorn.internal.runtime.ScriptFunctionData.invoke(ScriptFunctionData.java:637)
> at
> jdk.nashorn.internal.runtime.ScriptFunction.invoke(ScriptFunction.java:494)
> at
> jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:393)
> ... 10 more
> tez-build-tools/src/main/resources/checkstyle/suppressions.xml:
> java.lang.RuntimeException: java.io.FileNotFoundException:
> http://www.puppycrawl.com/dtds/suppressions_1_1.dtd
> at
> jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:397)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:449)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:406)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:402)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.eval(NashornScriptEngine.java:155)
> at
[jira] [Assigned] (TEZ-4212) Fix build checkstyle configuration and suppressions dtd URLs
[
https://issues.apache.org/jira/browse/TEZ-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles reassigned TEZ-4212:
---
Assignee: Jonathan Turner Eagles
> Fix build checkstyle configuration and suppressions dtd URLs
>
>
> Key: TEZ-4212
> URL: https://issues.apache.org/jira/browse/TEZ-4212
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Turner Eagles
>Assignee: Jonathan Turner Eagles
>Priority: Major
>
> {code}
> tez-build-tools/src/main/resources/checkstyle/checkstyle.xml:
> java.lang.RuntimeException: java.io.FileNotFoundException:
> http://www.puppycrawl.com/dtds/configuration_1_2.dtd
> at
> jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:397)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:449)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:406)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:402)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.eval(NashornScriptEngine.java:155)
> at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
> at com.sun.tools.script.shell.Main.evaluateString(Main.java:298)
> at com.sun.tools.script.shell.Main.evaluateString(Main.java:319)
> at com.sun.tools.script.shell.Main.access$300(Main.java:37)
> at com.sun.tools.script.shell.Main$3.run(Main.java:217)
> at com.sun.tools.script.shell.Main.main(Main.java:48)
> Caused by: java.io.FileNotFoundException:
> http://www.puppycrawl.com/dtds/configuration_1_2.dtd
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:647)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startEntity(XMLEntityManager.java:1304)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startDTDEntity(XMLEntityManager.java:1270)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.setInputSource(XMLDTDScannerImpl.java:264)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.dispatch(XMLDocumentScannerImpl.java:1161)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.next(XMLDocumentScannerImpl.java:1045)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:959)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
> at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at
> com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
> at
> com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
> at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:205)
> at
> jdk.nashorn.internal.scripts.Script$Recompilation$2$19313A$\^system_init\_.XMLDocument(:747)
> at jdk.nashorn.internal.scripts.Script$1$\^string\_.:program(:1)
> at
> jdk.nashorn.internal.runtime.ScriptFunctionData.invoke(ScriptFunctionData.java:637)
> at
> jdk.nashorn.internal.runtime.ScriptFunction.invoke(ScriptFunction.java:494)
> at
> jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:393)
> ... 10 more
> tez-build-tools/src/main/resources/checkstyle/suppressions.xml:
> java.lang.RuntimeException: java.io.FileNotFoundException:
> http://www.puppycrawl.com/dtds/suppressions_1_1.dtd
> at
> jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:397)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:449)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:406)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:402)
> at
> jdk.nashorn.api.scripting.NashornScriptEngine.eval(NashornScriptEngine.java:155)
> at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
> at
[jira] [Updated] (TEZ-4212) Fix build checkstyle configuration and suppressions dtd URLs
[
https://issues.apache.org/jira/browse/TEZ-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles updated TEZ-4212:
Description:
{code}
tez-build-tools/src/main/resources/checkstyle/checkstyle.xml:
java.lang.RuntimeException: java.io.FileNotFoundException:
http://www.puppycrawl.com/dtds/configuration_1_2.dtd
at
jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:397)
at
jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:449)
at
jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:406)
at
jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:402)
at
jdk.nashorn.api.scripting.NashornScriptEngine.eval(NashornScriptEngine.java:155)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
at com.sun.tools.script.shell.Main.evaluateString(Main.java:298)
at com.sun.tools.script.shell.Main.evaluateString(Main.java:319)
at com.sun.tools.script.shell.Main.access$300(Main.java:37)
at com.sun.tools.script.shell.Main$3.run(Main.java:217)
at com.sun.tools.script.shell.Main.main(Main.java:48)
Caused by: java.io.FileNotFoundException:
http://www.puppycrawl.com/dtds/configuration_1_2.dtd
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at
com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:647)
at
com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startEntity(XMLEntityManager.java:1304)
at
com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startDTDEntity(XMLEntityManager.java:1270)
at
com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.setInputSource(XMLDTDScannerImpl.java:264)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.dispatch(XMLDocumentScannerImpl.java:1161)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.next(XMLDocumentScannerImpl.java:1045)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:959)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at
com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
at
com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:205)
at
jdk.nashorn.internal.scripts.Script$Recompilation$2$19313A$\^system_init\_.XMLDocument(:747)
at jdk.nashorn.internal.scripts.Script$1$\^string\_.:program(:1)
at
jdk.nashorn.internal.runtime.ScriptFunctionData.invoke(ScriptFunctionData.java:637)
at
jdk.nashorn.internal.runtime.ScriptFunction.invoke(ScriptFunction.java:494)
at
jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:393)
... 10 more
tez-build-tools/src/main/resources/checkstyle/suppressions.xml:
java.lang.RuntimeException: java.io.FileNotFoundException:
http://www.puppycrawl.com/dtds/suppressions_1_1.dtd
at
jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:397)
at
jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:449)
at
jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:406)
at
jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:402)
at
jdk.nashorn.api.scripting.NashornScriptEngine.eval(NashornScriptEngine.java:155)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
at com.sun.tools.script.shell.Main.evaluateString(Main.java:298)
at com.sun.tools.script.shell.Main.evaluateString(Main.java:319)
at com.sun.tools.script.shell.Main.access$300(Main.java:37)
at com.sun.tools.script.shell.Main$3.run(Main.java:217)
at com.sun.tools.script.shell.Main.main(Main.java:48)
Caused by: java.io.FileNotFoundException:
http://www.puppycrawl.com/dtds/suppressions_1_1.dtd
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896)
[jira] [Created] (TEZ-4212) Fix build checkstyle configuration and suppressions dtd URLs
Jonathan Turner Eagles created TEZ-4212: --- Summary: Fix build checkstyle configuration and suppressions dtd URLs Key: TEZ-4212 URL: https://issues.apache.org/jira/browse/TEZ-4212 Project: Apache Tez Issue Type: Bug Reporter: Jonathan Turner Eagles -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4184) Tez-Build-Hadoop3 fails because of protoc 3.0.0
[
https://issues.apache.org/jira/browse/TEZ-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168127#comment-17168127
]
Jonathan Turner Eagles commented on TEZ-4184:
-
I have been working on this build this morning. I believe its purpose is to do
nightly full unit tests and yetus checks.
https://builds.apache.org/job/Tez-qbt-0.10-Build/
I'm still waiting for the results to declare success on this build.
> Tez-Build-Hadoop3 fails because of protoc 3.0.0
> ---
>
> Key: TEZ-4184
> URL: https://issues.apache.org/jira/browse/TEZ-4184
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Priority: Minor
>
> It builds as:
> {code}
> mvn clean test -DskipTests -Phadoop28 '-P!hadoop27'
> -Dhadoop.version=3.0.0-alpha2
> {code}
> https://builds.apache.org/job/Tez-Build-Hadoop3/250/console
> {code}
> [ERROR] Failed to execute goal
> org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2:protoc (compile-protoc)
> on project tez-api: org.apache.maven.plugin.MojoExecutionException: protoc
> version is 'libprotoc 3.0.0', expected version is '2.5.0' -> [Help 1]
> [ERROR]
> {code}
> I don't get the exact purpose of this job, but in the current way it doesn't
> make any sense, because hadoop3.x is still on protoc2.x until Hadoop 3.3, so
> the environment of this job should have protoc 2.5 installed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4204) Data race in RootInputInitializerManager
[ https://issues.apache.org/jira/browse/TEZ-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168087#comment-17168087 ] Mustafa Iman commented on TEZ-4204: --- [~abstractdog] since branch-0.9 does not contain TEZ-4170, this patch is not needed there. > Data race in RootInputInitializerManager > > > Key: TEZ-4204 > URL: https://issues.apache.org/jira/browse/TEZ-4204 > Project: Apache Tez > Issue Type: Bug >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Blocker > Attachments: TEZ-4204.1.patch, TEZ-4204.1.patch, TEZ-4204.2.patch > > > After https://issues.apache.org/jira/browse/TEZ-4170 there is a data race for > initializerMap in RootInputInitializerManager. initializerMap should be > initialized before vertex state is set to initializing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4204) Data race in RootInputInitializerManager
[ https://issues.apache.org/jira/browse/TEZ-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168065#comment-17168065 ] László Bodor commented on TEZ-4204: --- pushed to master, thanks [~mustafaiman]! also, I'm about to push it to branch-0.9 but found some conflicts while cherry-picking..could you please apply and attach a branch-0.9 version as you're more familiar with the patch? > Data race in RootInputInitializerManager > > > Key: TEZ-4204 > URL: https://issues.apache.org/jira/browse/TEZ-4204 > Project: Apache Tez > Issue Type: Bug >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Blocker > Attachments: TEZ-4204.1.patch, TEZ-4204.1.patch, TEZ-4204.2.patch > > > After https://issues.apache.org/jira/browse/TEZ-4170 there is a data race for > initializerMap in RootInputInitializerManager. initializerMap should be > initialized before vertex state is set to initializing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4206) TestSpeculation.testBasicSpeculationPerVertexConf is flaky
[ https://issues.apache.org/jira/browse/TEZ-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167792#comment-17167792 ] László Bodor commented on TEZ-4206: --- pushed to master and branch-0.9 thanks [~mustafaiman] for the patch and [~ashutoshc] for the review! > TestSpeculation.testBasicSpeculationPerVertexConf is flaky > -- > > Key: TEZ-4206 > URL: https://issues.apache.org/jira/browse/TEZ-4206 > Project: Apache Tez > Issue Type: Bug >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Attachments: TEZ-4206.1.patch > > > Test is flaky due to timing issue in MockDAGAppMaster's clock and > LegacySpeculator > [https://builds.apache.org/job/PreCommit-TEZ-Build/491/] > [https://builds.apache.org/job/PreCommit-TEZ-Build/492/] > [https://builds.apache.org/job/PreCommit-TEZ-Build/493/] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-4119) TestSpeculation is flaky
[ https://issues.apache.org/jira/browse/TEZ-4119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor resolved TEZ-4119. --- Resolution: Duplicate > TestSpeculation is flaky > > > Key: TEZ-4119 > URL: https://issues.apache.org/jira/browse/TEZ-4119 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: Ahmed Hussein >Priority: Major > Attachments: jstack.log, jstack4.log, jstack6.log, > org.apache.tez.dag.app.TestSpeculation-output.txt > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4184) Tez-Build-Hadoop3 fails because of protoc 3.0.0
[
https://issues.apache.org/jira/browse/TEZ-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167786#comment-17167786
]
László Bodor commented on TEZ-4184:
---
[~jeagles]: I can see this jenkins job is currently disabled, is this just a
temporary workaround or we won't need that anymore?
> Tez-Build-Hadoop3 fails because of protoc 3.0.0
> ---
>
> Key: TEZ-4184
> URL: https://issues.apache.org/jira/browse/TEZ-4184
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Priority: Minor
>
> It builds as:
> {code}
> mvn clean test -DskipTests -Phadoop28 '-P!hadoop27'
> -Dhadoop.version=3.0.0-alpha2
> {code}
> https://builds.apache.org/job/Tez-Build-Hadoop3/250/console
> {code}
> [ERROR] Failed to execute goal
> org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2:protoc (compile-protoc)
> on project tez-api: org.apache.maven.plugin.MojoExecutionException: protoc
> version is 'libprotoc 3.0.0', expected version is '2.5.0' -> [Help 1]
> [ERROR]
> {code}
> I don't get the exact purpose of this job, but in the current way it doesn't
> make any sense, because hadoop3.x is still on protoc2.x until Hadoop 3.3, so
> the environment of this job should have protoc 2.5 installed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4211) Optimise MergeManager final merge
[
https://issues.apache.org/jira/browse/TEZ-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167739#comment-17167739
]
TezQA commented on TEZ-4211:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
54s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
52s{color} | {color:red} tez-runtime-library in master has 1 extant findbugs
warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
17s{color} | {color:green} tez-runtime-library in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-TEZ-Build/504/artifact/out/Dockerfile |
| JIRA Issue | TEZ-4211 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13008734/TEZ-4211.wip.patch |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs
checkstyle compile |
| uname | Linux c3ba2501631f 4.15.0-101-generic #102-Ubuntu SMP Mon May 11
10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 2d7c60849 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| findbugs |
https://builds.apache.org/job/PreCommit-TEZ-Build/504/artifact/out/branch-findbugs-tez-runtime-library-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-TEZ-Build/504/testReport/ |
| Max. process+thread count | 126 (vs. ulimit of 5500) |
| modules | C: tez-runtime-library U: tez-runtime-library |
| Console output |
https://builds.apache.org/job/PreCommit-TEZ-Build/504/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Optimise MergeManager final merge
> -
>
> Key: TEZ-4211
> URL: https://issues.apache.org/jira/browse/TEZ-4211
> Project: Apache Tez
>
[jira] [Commented] (TEZ-4211) Optimise MergeManager final merge
[
https://issues.apache.org/jira/browse/TEZ-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167729#comment-17167729
]
Rajesh Balamohan commented on TEZ-4211:
---
Attaching wip patch
> Optimise MergeManager final merge
> -
>
> Key: TEZ-4211
> URL: https://issues.apache.org/jira/browse/TEZ-4211
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: TEZ-4211.wip.patch
>
>
> There are cases, when entire data is held in memory and no disk segments are
> present in MergeManager. Currently, mergemanager spills mem segments to disk
> before proceeding.
>
> [https://github.com/apache/tez/blob/master/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/MergeManager.java#L1184]
>
> {code:java}
> if (numMemDiskSegments > 0 && ioSortFactor > onDiskMapOutputs.size()) {
> ...
> ..
> TezMerger.writeFile(rIter, writer, progressable,
> TezRuntimeConfiguration.TEZ_RUNTIME_RECORDS_BEFORE_PROGRESS_DEFAULT);
> ...
> ..
> {code}
> This can be optimised not to spill to disk when only mem segments are present.
> Snippet from logs in one of the apps (Q78)
> {noformat}
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=839646500 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=859378362 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=856145179 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=849878734 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=842666749 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=839533127 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=860448335 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=844468505 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=850099810 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=849206236 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=840238680 and #onDiskOutputs=0,
> size=0
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (TEZ-4211) Optimise MergeManager final merge
[
https://issues.apache.org/jira/browse/TEZ-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-4211:
--
Attachment: TEZ-4211.wip.patch
> Optimise MergeManager final merge
> -
>
> Key: TEZ-4211
> URL: https://issues.apache.org/jira/browse/TEZ-4211
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: TEZ-4211.wip.patch
>
>
> There are cases, when entire data is held in memory and no disk segments are
> present in MergeManager. Currently, mergemanager spills mem segments to disk
> before proceeding.
>
> [https://github.com/apache/tez/blob/master/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/MergeManager.java#L1184]
>
> {code:java}
> if (numMemDiskSegments > 0 && ioSortFactor > onDiskMapOutputs.size()) {
> ...
> ..
> TezMerger.writeFile(rIter, writer, progressable,
> TezRuntimeConfiguration.TEZ_RUNTIME_RECORDS_BEFORE_PROGRESS_DEFAULT);
> ...
> ..
> {code}
> This can be optimised not to spill to disk when only mem segments are present.
> Snippet from logs in one of the apps (Q78)
> {noformat}
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=839646500 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=859378362 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=856145179 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=849878734 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=842666749 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=839533127 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=860448335 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=844468505 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=850099810 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=849206236 and #onDiskOutputs=0,
> size=0
> [ShuffleAndMergeRunner {Map_1} ()]
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
> finalMerge with #inMemoryOutputs=4112, size=840238680 and #onDiskOutputs=0,
> size=0
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[
https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167724#comment-17167724
]
László Bodor commented on TEZ-4175:
---
[~rajesh.balamohan]: could you please take a look at [^TEZ-4175.04.patch]?
I've validated this patch on a cluster (with hive on tez container mode),
worked fine
there were 2 things kept in mind:
1. use YarnConfiguration in TezYarnClient: double-checked and I've found that
YarnConfiguration [adds default
resources|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java#L100-L102],
and by using TezConfiguration everywhere, we can miss yarn config (as if
YarnConfiguration class is not referenced, we won't add yarn-*.xml as default
resource in its static initializer)
2. Use Configuration instead of YarnConfiguration in
DAGAppMaster...YarnConfiguration was added in TEZ-1879 with the
[comment|https://github.com/apache/tez/commit/ea46f459c0e88b0f9b0c714f8c6ac9d9a6f03c5e#diff-755c0ec043a1800cd6cbf31823a59c8fR1845],
I confirmed on the cluster that using a Configuration didn't cause any issues
> Consider removing YarnConfiguration where it's possible
> ---
>
> Key: TEZ-4175
> URL: https://issues.apache.org/jira/browse/TEZ-4175
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4175.01.patch, TEZ-4175.02.patch, TEZ-4175.03.patch,
> TEZ-4175.03.patch, TEZ-4175.04.patch
>
>
> A comment in DAGAppmaster made me think that we don't need to rely on
> [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java]
> in all cases, what if it can be replace with base Configuration...
> {code}
> // TODO Does this really need to be a YarnConfiguration ?
> Configuration conf = new Configuration(new YarnConfiguration());
> {code}
> In hadoop 3.1.3 source, I cannot see that it adds e.g. yarn-site as a
> resource:
> {code}
> public YarnConfiguration() {
> super();
> }
>
> public YarnConfiguration(Configuration conf) {
> super(conf);
> if (! (conf instanceof YarnConfiguration)) {
> this.reloadConfiguration();
> }
> }
> {code}
> in current codebase:
> {code}
> grep -iRH "new YarnConfiguration" --include="*.java"
> tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java:
> YarnConfiguration yarnConf = new YarnConfiguration(conf);
> tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:
> super.serviceInit(new YarnConfiguration(conf));
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/client/TestTezClient.java:
> tezClient.init(new TezConfiguration(false), new YarnConfiguration());
> tez-api/src/main/java/org/apache/tez/client/TezClient.java:
> amConfig.setYarnConfiguration(new
> YarnConfiguration(amConfig.getTezConfiguration()));
> tez-api/src/main/java/org/apache/tez/client/TezClient.java:
> amConfig.setYarnConfiguration(new
> YarnConfiguration(amConfig.getTezConfiguration()));
> tez-api/src/main/java/org/apache/tez/client/TezClient.java:return
> getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient,
> ugi);
> tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
> tezConf = new TezConfiguration(new YarnConfiguration());
> tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
>tezConf = new TezConfiguration(new YarnConfiguration(this.conf));
> tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java:
> Configuration testConf = new YarnConfiguration(
> tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java:
> this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf)));
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration conf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration conf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf =
[jira] [Created] (TEZ-4211) Optimise MergeManager final merge
Rajesh Balamohan created TEZ-4211:
-
Summary: Optimise MergeManager final merge
Key: TEZ-4211
URL: https://issues.apache.org/jira/browse/TEZ-4211
Project: Apache Tez
Issue Type: Bug
Reporter: Rajesh Balamohan
There are cases, when entire data is held in memory and no disk segments are
present in MergeManager. Currently, mergemanager spills mem segments to disk
before proceeding.
[https://github.com/apache/tez/blob/master/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/MergeManager.java#L1184]
{code:java}
if (numMemDiskSegments > 0 && ioSortFactor > onDiskMapOutputs.size()) {
...
..
TezMerger.writeFile(rIter, writer, progressable,
TezRuntimeConfiguration.TEZ_RUNTIME_RECORDS_BEFORE_PROGRESS_DEFAULT);
...
..
{code}
This can be optimised not to spill to disk when only mem segments are present.
Snippet from logs in one of the apps (Q78)
{noformat}
[ShuffleAndMergeRunner {Map_1} ()]
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
finalMerge with #inMemoryOutputs=4112, size=839646500 and #onDiskOutputs=0,
size=0
[ShuffleAndMergeRunner {Map_1} ()]
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
finalMerge with #inMemoryOutputs=4112, size=859378362 and #onDiskOutputs=0,
size=0
[ShuffleAndMergeRunner {Map_1} ()]
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
finalMerge with #inMemoryOutputs=4112, size=856145179 and #onDiskOutputs=0,
size=0
[ShuffleAndMergeRunner {Map_1} ()]
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
finalMerge with #inMemoryOutputs=4112, size=849878734 and #onDiskOutputs=0,
size=0
[ShuffleAndMergeRunner {Map_1} ()]
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
finalMerge with #inMemoryOutputs=4112, size=842666749 and #onDiskOutputs=0,
size=0
[ShuffleAndMergeRunner {Map_1} ()]
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
finalMerge with #inMemoryOutputs=4112, size=839533127 and #onDiskOutputs=0,
size=0
[ShuffleAndMergeRunner {Map_1} ()]
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
finalMerge with #inMemoryOutputs=4112, size=860448335 and #onDiskOutputs=0,
size=0
[ShuffleAndMergeRunner {Map_1} ()]
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
finalMerge with #inMemoryOutputs=4112, size=844468505 and #onDiskOutputs=0,
size=0
[ShuffleAndMergeRunner {Map_1} ()]
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
finalMerge with #inMemoryOutputs=4112, size=850099810 and #onDiskOutputs=0,
size=0
[ShuffleAndMergeRunner {Map_1} ()]
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
finalMerge with #inMemoryOutputs=4112, size=849206236 and #onDiskOutputs=0,
size=0
[ShuffleAndMergeRunner {Map_1} ()]
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager:
finalMerge with #inMemoryOutputs=4112, size=840238680 and #onDiskOutputs=0,
size=0
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
