[jira] [Created] (YARN-11658) ATS to make minimum HBase version 2.x
Steve Loughran created YARN-11658: - Summary: ATS to make minimum HBase version 2.x Key: YARN-11658 URL: https://issues.apache.org/jira/browse/YARN-11658 Project: Hadoop YARN Issue Type: Improvement Components: timelineserver Affects Versions: 3.4.0 Reporter: Steve Loughran following on from YARN-11657, what if we cut hbase 1.x support from ATS *entirely*? YARN-3 implies that the 2.x version might need to be bumped up -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved YARN-11657.
---
Fix Version/s: 3.3.9
3.4.1
3.5.0
Resolution: Fixed
> Remove protobuf-2.5 as dependency of hadoop-yarn-api
>
>
> Key: YARN-11657
> URL: https://issues.apache.org/jira/browse/YARN-11657
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.9, 3.4.1, 3.5.0
>
>
> hadoop-yarn-api is still exporting protobuf-2.5.
> if we can cut this, we should
> {code}
> [echo] [INFO] +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
> [echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
> for duplicate)
> [echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
> omitted for duplicate)
> [echo] [INFO] | | +-
> (org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for
> duplicate)
> [echo] [INFO] | | +-
> com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
> omitted for duplicate)
> [echo] [INFO] | | \-
> (com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
> for duplicate)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
Steve Loughran created YARN-11657:
-
Summary: Remove protobuf-2.5 as dependency of hadoop-yarn-api
Key: YARN-11657
URL: https://issues.apache.org/jira/browse/YARN-11657
Project: Hadoop YARN
Issue Type: Improvement
Components: api
Affects Versions: 3.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
hadoop-yarn-api is still exporting protobuf-2.5.
if we can cut this, we should
{code}
[echo] [INFO] +-
org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
[echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
[echo] [INFO] | | +-
(org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
for duplicate)
[echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
omitted for duplicate)
[echo] [INFO] | | +-
(org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for duplicate)
[echo] [INFO] | | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[echo] [INFO] | | +-
(org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
omitted for duplicate)
[echo] [INFO] | | \-
(com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
for duplicate)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11498) Exclude Jettison from jersey-json artifact in hadoop-yarn-common's pom.xml
[ https://issues.apache.org/jira/browse/YARN-11498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11498: -- Fix Version/s: 3.3.9 > Exclude Jettison from jersey-json artifact in hadoop-yarn-common's pom.xml > -- > > Key: YARN-11498 > URL: https://issues.apache.org/jira/browse/YARN-11498 > Project: Hadoop YARN > Issue Type: Task > Components: build >Reporter: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > This exclusion is done to reduce CVEs present due to an older version of > Jettison(1.1) being pulled in with jersey-json artifact. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11546) NPE in TestTimelineAuthFilterForV2
[ https://issues.apache.org/jira/browse/YARN-11546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778661#comment-17778661 ] Steve Loughran commented on YARN-11546: --- ..spotted again. i think it may actually be in the finally() clause which calls close()...if the reader is null this will hide any underlying exception triggered trying to read the file > NPE in TestTimelineAuthFilterForV2 > -- > > Key: YARN-11546 > URL: https://issues.apache.org/jira/browse/YARN-11546 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Minor > > NPE in some yetus tests > ``` > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2.readEntityFile(TestTimelineAuthFilterForV2.java:314) > at > org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2.verifyEntity(TestTimelineAuthFilterForV2.java:287) > at > org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2.testPutTimelineEntities(TestTimelineAuthFilterForV2.java:441) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62 > ``` > the actual line is reader.close() in finally; this will trigger an NPE if > reader == null, i.e. file opening failed. > Something else has gone wrong, but this NPE is hiding it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Moved] (YARN-11569) compile failure due to nodejs is incompatible with module
[ https://issues.apache.org/jira/browse/YARN-11569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran moved HADOOP-18892 to YARN-11569: Component/s: build (was: build) Key: YARN-11569 (was: HADOOP-18892) Affects Version/s: 3.3.6 (was: 3.3.6) Project: Hadoop YARN (was: Hadoop Common) > compile failure due to nodejs is incompatible with module > - > > Key: YARN-11569 > URL: https://issues.apache.org/jira/browse/YARN-11569 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.3.6 > Environment: hadoop 3.3.6 >Reporter: Authur Wang >Priority: Critical > Labels: pull-request-available > > hadoop compile error due to the following error: > [INFO] Running 'yarn ' in > /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-6058/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target > [INFO] yarn install v1.22.5 > [INFO] info No lockfile found. > [INFO] [1/4] Resolving packages... > [INFO] warning angular-route@1.6.10: For the actively supported Angular, see > https://www.npmjs.com/package/@angular/core. AngularJS support has officially > ended. For extended AngularJS support options, see > https://goo.gle/angularjs-path-forward. > [INFO] warning angular@1.6.10: For the actively supported Angular, see > https://www.npmjs.com/package/@angular/core. AngularJS support has officially > ended. For extended AngularJS support options, see > https://goo.gle/angularjs-path-forward. > [INFO] [2/4] Fetching packages... > [INFO] error triple-beam@1.4.1: The engine "node" is incompatible with this > module. Expected version ">= 14.0.0". Got "12.22.1" > [INFO] error Found incompatible module. > [INFO] error Found incompatible module.info Visit > https://yarnpkg.com/en/docs/cli/install for documentation about this command. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11568) hadoop distcp needs support to filter by file/directory attribute
[ https://issues.apache.org/jira/browse/YARN-11568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764557#comment-17764557 ] Steve Loughran commented on YARN-11568: --- ok. you can just use the "move issue" item to reassign things in future > hadoop distcp needs support to filter by file/directory attribute > - > > Key: YARN-11568 > URL: https://issues.apache.org/jira/browse/YARN-11568 > Project: Hadoop YARN > Issue Type: New Feature > Components: applications >Affects Versions: 3.3.6 >Reporter: Authur Wang >Priority: Minor > Fix For: 3.3.6 > > > In some circumstances, we need to filter file/directory by file/directroy. > For example, we need to filter out them by file modified time, isDir attrs, > etc. > So, should we introduce a new method public boolean > shouldCopy(CopyListingFileStatus fileStatus). by this method, we can > introduce a more fluent way to do things than public abstract boolean > shouldCopy(Path path). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10992) schema generation produces code which doesn't compile on java 11+
[ https://issues.apache.org/jira/browse/YARN-10992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved YARN-10992. --- Resolution: Cannot Reproduce trunk seems to build on java17 now... > schema generation produces code which doesn't compile on java 11+ > - > > Key: YARN-10992 > URL: https://issues.apache.org/jira/browse/YARN-10992 > Project: Hadoop YARN > Issue Type: Sub-task > Components: buid >Affects Versions: 3.4.0 > Environment: build with openjdk 17 >Reporter: Steve Loughran >Priority: Major > > The changes of YARN-10386 are somehow stopping java 17 (maybe 11_+?) > compilation, as the generated classes have an import > javax.annotation.Generated which is not in the JRE classes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11546) NPE in TestTimelineAuthFilterForV2
Steve Loughran created YARN-11546: - Summary: NPE in TestTimelineAuthFilterForV2 Key: YARN-11546 URL: https://issues.apache.org/jira/browse/YARN-11546 Project: Hadoop YARN Issue Type: Bug Components: test Affects Versions: 3.4.0 Reporter: Steve Loughran NPE in some yetus tests ``` java.lang.NullPointerException at org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2.readEntityFile(TestTimelineAuthFilterForV2.java:314) at org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2.verifyEntity(TestTimelineAuthFilterForV2.java:287) at org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2.testPutTimelineEntities(TestTimelineAuthFilterForV2.java:441) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62 ``` the actual line is reader.close() in finally; this will trigger an NPE if reader == null, i.e. file opening failed. Something else has gone wrong, but this NPE is hiding it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11535) Snakeyaml should be excluded from jackson-dataformat-yaml-2.12.7 as it may cause transitive dependency issue.
[ https://issues.apache.org/jira/browse/YARN-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744658#comment-17744658 ] Steve Loughran commented on YARN-11535: --- can you tag with which versions are affected, thanks > Snakeyaml should be excluded from jackson-dataformat-yaml-2.12.7 as it may > cause transitive dependency issue. > - > > Key: YARN-11535 > URL: https://issues.apache.org/jira/browse/YARN-11535 > Project: Hadoop YARN > Issue Type: Task > Components: build, yarn >Reporter: Susheel Gupta >Priority: Major > > Hadoop-project uses > [snakeyaml.version-2.0|https://github.com/apache/hadoop/blame/trunk/hadoop-project/pom.xml#L198] > and > [jackson-dataformat-yaml-2.12.7|https://github.com/apache/hadoop/blob/trunk/hadoop-project/pom.xml#L72]. > But internally jackson-dataformat-yaml-2.12.7 uses compile dependency > [snakeyaml.version-1.27|https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-yaml/2.12.7] > .This may cause a transitive dependency issue in other services using hadoop > jar having jackson-dataformat-yaml-2.12.7 as jackson-dataformat-yaml-2.12.7 > will use nearest dependency available of snakeyaml i.e 1.27 and ignore the > version of snakeyaml-2.0 from hadoop-project. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11535) Snakeyaml should be excluded from jackson-dataformat-yaml-2.12.7 as it may cause transitive dependency issue.
[ https://issues.apache.org/jira/browse/YARN-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11535: -- Component/s: build > Snakeyaml should be excluded from jackson-dataformat-yaml-2.12.7 as it may > cause transitive dependency issue. > - > > Key: YARN-11535 > URL: https://issues.apache.org/jira/browse/YARN-11535 > Project: Hadoop YARN > Issue Type: Task > Components: build, yarn >Reporter: Susheel Gupta >Priority: Major > > Hadoop-project uses > [snakeyaml.version-2.0|https://github.com/apache/hadoop/blame/trunk/hadoop-project/pom.xml#L198] > and > [jackson-dataformat-yaml-2.12.7|https://github.com/apache/hadoop/blob/trunk/hadoop-project/pom.xml#L72]. > But internally jackson-dataformat-yaml-2.12.7 uses compile dependency > [snakeyaml.version-1.27|https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-yaml/2.12.7] > .This may cause a transitive dependency issue in other services using hadoop > jar having jackson-dataformat-yaml-2.12.7 as jackson-dataformat-yaml-2.12.7 > will use nearest dependency available of snakeyaml i.e 1.27 and ignore the > version of snakeyaml-2.0 from hadoop-project. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11458) Maven parallel build fails when Yarn UI v2 is enabled
[
https://issues.apache.org/jira/browse/YARN-11458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-11458:
--
Target Version/s: 3.4.0, 3.3.9 (was: 3.4.0, 3.3.5, 3.3.4, 3.3.9)
> Maven parallel build fails when Yarn UI v2 is enabled
> -
>
> Key: YARN-11458
> URL: https://issues.apache.org/jira/browse/YARN-11458
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn, yarn-ui-v2
>Affects Versions: 3.4.0, 3.3.5, 3.3.4, 3.3.9
> Environment: The problem occurs sporadically while using the Hadoop
> development environment (Ubuntu)
>Reporter: Steve Vaughan
>Assignee: Steve Vaughan
>Priority: Critical
>
> Running a parallel build fails during assembly with the following error when
> running either package or install:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.4:single (package-yarn) on
> project hadoop-yarn-project: Failed to create assembly: Error creating
> assembly archive hadoop-yarn-dist:
> /workspace/hadoop-yarn-project/./hadoop-yarn/hadoop-yarn-ui/target/webapp/tmp/sass_compiler-input_base_path-iNYf9pEm.tmp/vendor/ember-qunit/ember-qunit.map
> -> [Help 1]{code}
> This appears to be a race condition introduce when `-Pyarn-ui` is used
> because the `hadoop-yarn-project` does not have a dependency listed for
> `yarn-ui`.
> The command executed was:
> {code:java}
> $ mvn -nsu clean install -Pdist,native -DskipTests -Dtar
> -Dmaven.javadoc.skip=true -T 2C {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11394) Fix hadoop-yarn-server-resourcemanager module Java Doc Errors.
[ https://issues.apache.org/jira/browse/YARN-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11394: -- Fix Version/s: 3.3.9 > Fix hadoop-yarn-server-resourcemanager module Java Doc Errors. > -- > > Key: YARN-11394 > URL: https://issues.apache.org/jira/browse/YARN-11394 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > When I finished the pr of YARN-11350, I encountered java doc error, this jira > will fix the java doc error in hadoop-yarn-server-resourcemanager. > > The java doc error report is as follows: > https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5131/22/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11453) Update solr from 8.8.2 to 8.11.2
[
https://issues.apache.org/jira/browse/YARN-11453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698877#comment-17698877
]
Steve Loughran commented on YARN-11453:
---
not any code I go near; sorry. chase it up on the yarn dev list
> Update solr from 8.8.2 to 8.11.2
>
>
> Key: YARN-11453
> URL: https://issues.apache.org/jira/browse/YARN-11453
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: applications
>Affects Versions: 3.3.4
>Reporter: Xuesen Liang
>Priority: Minor
> Labels: pull-request-available
>
> Solr is used in hadoop-yarn-applications-catalog-webapp.
> {code:sh}
> $ grep solr.version . -r
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml:
> ${solr.version}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml:
> ${solr.version}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml:
> ${solr.version}
> ./hadoop-project/pom.xml: 8.8.2
> $
> {code}
>
> Solr-8.8.2's log4j2 version is 2.13.2.
> Solr-8.11.2's log4j2 version is 2.17.1 (
> https://issues.apache.org/jira/browse/SOLR-15871 )
> Should we update solr to 8.11.2 correspondingly?
> Then maven will not try to download log4j2-2.13.2, which is forbidden by some
> maven repository.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11454) high-entropy abfs/https proxy passwords don't make it through to yarn/mr worker processes
Steve Loughran created YARN-11454: - Summary: high-entropy abfs/https proxy passwords don't make it through to yarn/mr worker processes Key: YARN-11454 URL: https://issues.apache.org/jira/browse/YARN-11454 Project: Hadoop YARN Issue Type: Bug Components: client, yarn Affects Versions: 3.3.4 Reporter: Steve Loughran Assignee: Steve Loughran Fun one. ABFS proxy settings come through sysprops not hadoop conf options, so have to bse set in one of the various .java.opts config options -for am and workers. Except with all the layers in the way, there are some corner cases of char which don't make it through. And the default solution "don't do that" comes up against central IT policies of "no can do" Fix: work out what the brittle chars are escape so that launcher doesn't get the clever idea to join in. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11454) high-entropy abfs/https proxy passwords don't make it through to yarn/mr worker processes
[ https://issues.apache.org/jira/browse/YARN-11454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17697385#comment-17697385 ] Steve Loughran commented on YARN-11454: --- note, we do want env var eval on the launcher cli > high-entropy abfs/https proxy passwords don't make it through to yarn/mr > worker processes > - > > Key: YARN-11454 > URL: https://issues.apache.org/jira/browse/YARN-11454 > Project: Hadoop YARN > Issue Type: Bug > Components: client, yarn >Affects Versions: 3.3.4 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > > Fun one. ABFS proxy settings come through sysprops not hadoop conf options, > so have to bse set in one of the various .java.opts config options -for am > and workers. > Except with all the layers in the way, there are some corner cases of char > which don't make it through. And the default solution "don't do that" comes > up against central IT policies of "no can do" > Fix: work out what the brittle chars are escape so that launcher doesn't get > the clever idea to join in. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10444) use openFile() with sequential IO for localizing files.
[ https://issues.apache.org/jira/browse/YARN-10444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved YARN-10444. --- Resolution: Fixed came in with HADOOP-16202 > use openFile() with sequential IO for localizing files. > --- > > Key: YARN-10444 > URL: https://issues.apache.org/jira/browse/YARN-10444 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.3.5 > > > HADOOP-16202 adds standard options for declaring the read/seek > Policy when reading a file. These should be set to sequential IO > When localising resources, so that if the default/cluster settings > For a file system are optimized for random IO, artifact downloads > are still read at the maximum speed possible (one big GET to the EOF). > Most of this happens in hadoop-common, but some tuning of FSDownload > can assist > * tar/jar download must also be sequential > * if the FileStatus is passed around, that can be used > in the open request to skip checks when loading the file. > > Together this can save 3 HEAD requests per resource, with the sequential > IO avoiding any splitting of the big read into separate block GETs -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10444) Node Manager to use openFile() with whole-file read policy for localizing files.
[ https://issues.apache.org/jira/browse/YARN-10444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-10444: -- Summary: Node Manager to use openFile() with whole-file read policy for localizing files. (was: use openFile() with sequential IO for localizing files.) > Node Manager to use openFile() with whole-file read policy for localizing > files. > > > Key: YARN-10444 > URL: https://issues.apache.org/jira/browse/YARN-10444 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.3.5 > > > HADOOP-16202 adds standard options for declaring the read/seek > Policy when reading a file. These should be set to sequential IO > When localising resources, so that if the default/cluster settings > For a file system are optimized for random IO, artifact downloads > are still read at the maximum speed possible (one big GET to the EOF). > Most of this happens in hadoop-common, but some tuning of FSDownload > can assist > * tar/jar download must also be sequential > * if the FileStatus is passed around, that can be used > in the open request to skip checks when loading the file. > > Together this can save 3 HEAD requests per resource, with the sequential > IO avoiding any splitting of the big read into separate block GETs -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10444) use openFile() with sequential IO for localizing files.
[ https://issues.apache.org/jira/browse/YARN-10444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-10444: -- Fix Version/s: 3.3.5 > use openFile() with sequential IO for localizing files. > --- > > Key: YARN-10444 > URL: https://issues.apache.org/jira/browse/YARN-10444 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.3.5 > > > HADOOP-16202 adds standard options for declaring the read/seek > Policy when reading a file. These should be set to sequential IO > When localising resources, so that if the default/cluster settings > For a file system are optimized for random IO, artifact downloads > are still read at the maximum speed possible (one big GET to the EOF). > Most of this happens in hadoop-common, but some tuning of FSDownload > can assist > * tar/jar download must also be sequential > * if the FileStatus is passed around, that can be used > in the open request to skip checks when loading the file. > > Together this can save 3 HEAD requests per resource, with the sequential > IO avoiding any splitting of the big read into separate block GETs -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11394) Fix hadoop-yarn-server-resourcemanager module Java Doc Errors.
[ https://issues.apache.org/jira/browse/YARN-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved YARN-11394. --- Fix Version/s: 3.4.0 Resolution: Fixed > Fix hadoop-yarn-server-resourcemanager module Java Doc Errors. > -- > > Key: YARN-11394 > URL: https://issues.apache.org/jira/browse/YARN-11394 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > When I finished the pr of YARN-11350, I encountered java doc error, this jira > will fix the java doc error in hadoop-yarn-server-resourcemanager. > > The java doc error report is as follows: > https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5131/22/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11441) Revert YARN-10495
[ https://issues.apache.org/jira/browse/YARN-11441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved YARN-11441. --- Fix Version/s: 3.4.0 3.3.5 Resolution: Done > Revert YARN-10495 > - > > Key: YARN-11441 > URL: https://issues.apache.org/jira/browse/YARN-11441 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Trivial > Fix For: 3.4.0, 3.3.5 > > > roll back YARN-10495 as it is causing problems on some deployments; creating > a new jira for due diligence. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11441) Revert YARN-10495
Steve Loughran created YARN-11441: - Summary: Revert YARN-10495 Key: YARN-11441 URL: https://issues.apache.org/jira/browse/YARN-11441 Project: Hadoop YARN Issue Type: Task Components: yarn Affects Versions: 3.3.5 Reporter: Steve Loughran Assignee: Steve Loughran roll back YARN-10495 as it is causing problems on some deployments; creating a new jira for due diligence. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11397) Memory leak when reading aggregated logs from s3 (LogAggregationTFileController::readAggregatedLogs)
[ https://issues.apache.org/jira/browse/YARN-11397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648993#comment-17648993 ] Steve Loughran commented on YARN-11397: --- both issues are to be fixed in 3.3.5 -once that RC comes out please test to make sure it works for you. > Memory leak when reading aggregated logs from s3 > (LogAggregationTFileController::readAggregatedLogs) > > > Key: YARN-11397 > URL: https://issues.apache.org/jira/browse/YARN-11397 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation >Affects Versions: 3.2.2 > Environment: Remote logs dir on s3. >Reporter: Maciej Smolenski >Priority: Critical > Attachments: YarnLogsS3Issue.scala > > > Reproduction code in the attachment. > When collecting aggregated logs from s3 in a loop (see reproduction code) we > can easily see that the number of 'S3AInstrumentation' is increasing although > the number of 'S3AFileSystem' is not increasing. It means that > 'S3AInstrumentation' is not released together with 'S3AFileSystem' as it > should be. The root cause of this seems to be the missing close on > S3AFileSystem. > The issue seems similar to https://issues.apache.org/jira/browse/YARN-11039 > but the issue is a 'memory leak' (not a 'thread leak') and affected version > is earlier here (3.2.2). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11397) Memory leak when reading aggregated logs from s3 (LogAggregationTFileController::readAggregatedLogs)
[ https://issues.apache.org/jira/browse/YARN-11397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648992#comment-17648992 ] Steve Loughran commented on YARN-11397: --- root cause is that FileContext doesn't have a close() so willl hang on to s3a refs with calling close() on them. this can leak thread pools, and if not that, the strong back references from metrics. HADOOP-18476 *probably* fixes the thread pool issue, while HADOOP-18526 will address the instrumentation leaks. i think really we should make FileContext closeable and do a full end-to-end close there, then call close() in our app. The s3a changes are very much damage limitation. > Memory leak when reading aggregated logs from s3 > (LogAggregationTFileController::readAggregatedLogs) > > > Key: YARN-11397 > URL: https://issues.apache.org/jira/browse/YARN-11397 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation >Affects Versions: 3.2.2 > Environment: Remote logs dir on s3. >Reporter: Maciej Smolenski >Priority: Critical > Attachments: YarnLogsS3Issue.scala > > > Reproduction code in the attachment. > When collecting aggregated logs from s3 in a loop (see reproduction code) we > can easily see that the number of 'S3AInstrumentation' is increasing although > the number of 'S3AFileSystem' is not increasing. It means that > 'S3AInstrumentation' is not released together with 'S3AFileSystem' as it > should be. The root cause of this seems to be the missing close on > S3AFileSystem. > The issue seems similar to https://issues.apache.org/jira/browse/YARN-11039 > but the issue is a 'memory leak' (not a 'thread leak') and affected version > is earlier here (3.2.2). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10992) schema generation produces code which doesn't compile on java 11+
[
https://issues.apache.org/jira/browse/YARN-10992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17647501#comment-17647501
]
Steve Loughran commented on YARN-10992:
---
update.
# the maven pojo plugin generates code with the correct @Generated attribute
for that java version
# the plugin doesn't regenerate the .java classes if they are present
# if you have done a build with java11, you can't go back to java 8
fix
`
{code}
rm
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/schema/*
{code}
maybe the mvn clean operation should do this. If the classes were generated
into target/generated/java this would be automatic
> schema generation produces code which doesn't compile on java 11+
> -
>
> Key: YARN-10992
> URL: https://issues.apache.org/jira/browse/YARN-10992
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: buid
>Affects Versions: 3.4.0
> Environment: build with openjdk 17
>Reporter: Steve Loughran
>Priority: Major
>
> The changes of YARN-10386 are somehow stopping java 17 (maybe 11_+?)
> compilation, as the generated classes have an import
> javax.annotation.Generated which is not in the JRE classes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11384) NPE in DelegationTokenRenewer causes all subsequent apps to fail with "Timer already cancelled"
[
https://issues.apache.org/jira/browse/YARN-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17645283#comment-17645283
]
Steve Loughran commented on YARN-11384:
---
looks like YARN-9109; same stack trace just different cause
> NPE in DelegationTokenRenewer causes all subsequent apps to fail with "Timer
> already cancelled"
> ---
>
> Key: YARN-11384
> URL: https://issues.apache.org/jira/browse/YARN-11384
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Aditya Sharma
>Assignee: Aditya Sharma
>Priority: Major
>
> All newly submitted yarn apps start failing with following error in the
> diagnostic message
> {noformat}
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Unable to add the application to the delegation token renewer.
> java.lang.IllegalStateException: Timer already cancelled.
> at java.util.Timer.sched(Timer.java:397)
> at java.util.Timer.schedule(Timer.java:208)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.setTimerForTokenRenewal(DelegationTokenRenewer.java:604)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:515)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$900(DelegationTokenRenewer.java:79)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:923)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:900)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> Any uncaught exception at DelegationTokenRenewer.RenewalTimerTask#run causes
> all subsequent yarn apps to fail with java.lang.IllegalStateException: Timer
> already cancelled
> One such NPE is thrown when DelegationTokenRenewer.RenewalTimerTask#run
> invokes DelegationTokenRenewer#removeFailedDelegationToken that tries to
> remove token from the "DelegationTokenRenewer#appTokens" for an applicationId.
> If DelegationTokenRenewer#appTokens map didn’t have the Set> entry while token (DelegationTokenToRenew) had
> the reference to the applicationId, then NPE is thrown leading to "Timer
> already cancelled"
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11389) Upgrade spring-core to 5.3.20 in wro4j-maven-plugin
[ https://issues.apache.org/jira/browse/YARN-11389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11389: -- Labels: transitive-cve (was: ) > Upgrade spring-core to 5.3.20 in wro4j-maven-plugin > > > Key: YARN-11389 > URL: https://issues.apache.org/jira/browse/YARN-11389 > Project: Hadoop YARN > Issue Type: Improvement > Components: buid, yarn-ui-v2 >Affects Versions: 3.4.0 >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Minor > Labels: transitive-cve > > Currently during yarn-ui build we are using vulnerable version of > spring-core-3.1.1.RELEASE.jar which has serveral critical and high > vulnerablilites, we need to upgrade to a version 5.3.20+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11389) Upgrade spring-core to 5.3.20 in wro4j-maven-plugin
[ https://issues.apache.org/jira/browse/YARN-11389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11389: -- Component/s: buid > Upgrade spring-core to 5.3.20 in wro4j-maven-plugin > > > Key: YARN-11389 > URL: https://issues.apache.org/jira/browse/YARN-11389 > Project: Hadoop YARN > Issue Type: Improvement > Components: buid, yarn-ui-v2 >Affects Versions: 3.4.0 >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Minor > > Currently during yarn-ui build we are using vulnerable version of > spring-core-3.1.1.RELEASE.jar which has serveral critical and high > vulnerablilites, we need to upgrade to a version 5.3.20+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11389) Upgrade spring-core to 5.3.20 in wro4j-maven-plugin
[ https://issues.apache.org/jira/browse/YARN-11389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17643769#comment-17643769 ] Steve Loughran commented on YARN-11389: --- this is build, not shipping; i've just verified that in the 3.3.5 rc and a recent trunk build. downgrading to minor, marking as "build". > Upgrade spring-core to 5.3.20 in wro4j-maven-plugin > > > Key: YARN-11389 > URL: https://issues.apache.org/jira/browse/YARN-11389 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn-ui-v2 >Affects Versions: 3.4.0 >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > > Currently during yarn-ui build we are using vulnerable version of > spring-core-3.1.1.RELEASE.jar which has serveral critical and high > vulnerablilites, we need to upgrade to a version 5.3.20+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11389) Upgrade spring-core to 5.3.20 in wro4j-maven-plugin
[ https://issues.apache.org/jira/browse/YARN-11389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11389: -- Priority: Minor (was: Major) > Upgrade spring-core to 5.3.20 in wro4j-maven-plugin > > > Key: YARN-11389 > URL: https://issues.apache.org/jira/browse/YARN-11389 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn-ui-v2 >Affects Versions: 3.4.0 >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Minor > > Currently during yarn-ui build we are using vulnerable version of > spring-core-3.1.1.RELEASE.jar which has serveral critical and high > vulnerablilites, we need to upgrade to a version 5.3.20+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11267) Upgrade JUnit from 4 to 5 in hadoop-yarn-server-router
[ https://issues.apache.org/jira/browse/YARN-11267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17643419#comment-17643419 ] Steve Loughran commented on YARN-11267: --- ashtosh. don't set fix version to anything unless committed; we need that for tracking where things are in and release notes. Use Target Version to say where you want things to go, all of which must be future releases. thanks. > Upgrade JUnit from 4 to 5 in hadoop-yarn-server-router > -- > > Key: YARN-11267 > URL: https://issues.apache.org/jira/browse/YARN-11267 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test, yarn >Affects Versions: 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11267) Upgrade JUnit from 4 to 5 in hadoop-yarn-server-router
[ https://issues.apache.org/jira/browse/YARN-11267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11267: -- Fix Version/s: (was: 3.3.5) > Upgrade JUnit from 4 to 5 in hadoop-yarn-server-router > -- > > Key: YARN-11267 > URL: https://issues.apache.org/jira/browse/YARN-11267 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test, yarn >Affects Versions: 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11266) Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase-tests
[ https://issues.apache.org/jira/browse/YARN-11266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17643418#comment-17643418 ] Steve Loughran commented on YARN-11266: --- ashtosh. don't set fix version to anything unless committed; we need that for tracking where things are in and release notes. Use Target Version to say where you want things to go, all of which must be future releases. thanks. > Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase-tests > --- > > Key: YARN-11266 > URL: https://issues.apache.org/jira/browse/YARN-11266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test, yarn >Affects Versions: 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11266) Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase-tests
[ https://issues.apache.org/jira/browse/YARN-11266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11266: -- Fix Version/s: (was: 3.3.5) > Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase-tests > --- > > Key: YARN-11266 > URL: https://issues.apache.org/jira/browse/YARN-11266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test, yarn >Affects Versions: 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11330) Use secure XML parser utils in YARN
[ https://issues.apache.org/jira/browse/YARN-11330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11330: -- Description: Uptakes HADOOP-18469 If anyone is landing on this page following any security scanner alert, know that there is no known issue here, just a centralisation of all construction of XML parsers with lockdown of all the features. was:Uptakes HADOOP-18469 > Use secure XML parser utils in YARN > --- > > Key: YARN-11330 > URL: https://issues.apache.org/jira/browse/YARN-11330 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.4 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > > Uptakes HADOOP-18469 > If anyone is landing on this page following any security scanner alert, know > that there is no known issue here, just a centralisation of all construction > of XML parsers with lockdown of all the features. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11330) Use secure XML parser utils in YARN
[ https://issues.apache.org/jira/browse/YARN-11330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved YARN-11330. --- Fix Version/s: 3.4.0 3.3.5 Resolution: Fixed > Use secure XML parser utils in YARN > --- > > Key: YARN-11330 > URL: https://issues.apache.org/jira/browse/YARN-11330 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.4 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > > Uptakes HADOOP-18469 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11039) LogAggregationFileControllerFactory::getFileControllerForRead can leak threads
[
https://issues.apache.org/jira/browse/YARN-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved YARN-11039.
---
Fix Version/s: 3.4.0
3.3.5
Resolution: Fixed
merged in the hadoop patch HADOOP-18476 which fixes this
> LogAggregationFileControllerFactory::getFileControllerForRead can leak
> threads
> ---
>
> Key: YARN-11039
> URL: https://issues.apache.org/jira/browse/YARN-11039
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.3.4
>Reporter: Rajesh Balamohan
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: performance, stability
> Fix For: 3.4.0, 3.3.5
>
>
> getFileControllerForRead::getFileControllerForRead internally opens up a new
> FS object everytime and is not closed.
> When cloud connectors (e.g s3a) is used along with Knox, it ends up leaking
> KnoxTokenMonitor for every unclosed FS object causing thread leaks in NM.
> Lines of interest:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileControllerFactory.java#L167]
> {noformat}
>try {
> Path remoteAppLogDir = fileController.getOlderRemoteAppLogDir(appId,
> appOwner);
> if (LogAggregationUtils.getNodeFiles(conf, remoteAppLogDir, appId,
> appOwner).hasNext()) {
> return fileController;
> }
> } catch (Exception ex) {
> diagnosticsMsg.append(ex.getMessage() + "\n");
> continue;
> }
> {noformat}
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationUtils.java#L252]
> {noformat}
>
> public static RemoteIterator getNodeFiles(Configuration conf,
> Path remoteAppLogDir, ApplicationId appId, String appOwner)
> throws IOException {
> Path qualifiedLogDir =
> FileContext.getFileContext(conf).makeQualified(remoteAppLogDir);
> return FileContext.getFileContext(
> qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir);
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11330) Use secure XML parser utils in YARN
[ https://issues.apache.org/jira/browse/YARN-11330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned YARN-11330: - Assignee: PJ Fanning (was: Steve Loughran) > Use secure XML parser utils in YARN > --- > > Key: YARN-11330 > URL: https://issues.apache.org/jira/browse/YARN-11330 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.4 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > > Uptakes HADOOP-18469 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11330) Use secure XML parser utils in YARN
[ https://issues.apache.org/jira/browse/YARN-11330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned YARN-11330: - Assignee: Steve Loughran > Use secure XML parser utils in YARN > --- > > Key: YARN-11330 > URL: https://issues.apache.org/jira/browse/YARN-11330 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.4 >Reporter: PJ Fanning >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > Uptakes HADOOP-18469 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11330) Use secure XML parser utils in YARN
[ https://issues.apache.org/jira/browse/YARN-11330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11330: -- Summary: Use secure XML parser utils in YARN (was: Use secure XML parser utils) > Use secure XML parser utils in YARN > --- > > Key: YARN-11330 > URL: https://issues.apache.org/jira/browse/YARN-11330 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: PJ Fanning >Priority: Major > Labels: pull-request-available > > Uptakes HADOOP-18469 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11330) Use secure XML parser utils in YARN
[ https://issues.apache.org/jira/browse/YARN-11330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11330: -- Affects Version/s: 3.3.4 > Use secure XML parser utils in YARN > --- > > Key: YARN-11330 > URL: https://issues.apache.org/jira/browse/YARN-11330 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.4 >Reporter: PJ Fanning >Priority: Major > Labels: pull-request-available > > Uptakes HADOOP-18469 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11303) Upgrade jquery ui to 1.13.2
[ https://issues.apache.org/jira/browse/YARN-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11303: -- Component/s: security > Upgrade jquery ui to 1.13.2 > --- > > Key: YARN-11303 > URL: https://issues.apache.org/jira/browse/YARN-11303 > Project: Hadoop YARN > Issue Type: Improvement > Components: security >Reporter: D M Murali Krishna Reddy >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The current jquery-ui version used(1.13.1) in the trunk has the following > vulnerability > [CVE-2022-31160|https://nvd.nist.gov/vuln/detail/CVE-2022-31160] so we need > to upgrade to at least 1.13.2. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11303) Upgrade jquery ui to 1.13.2
[ https://issues.apache.org/jira/browse/YARN-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11303: -- Fix Version/s: 3.3.5 > Upgrade jquery ui to 1.13.2 > --- > > Key: YARN-11303 > URL: https://issues.apache.org/jira/browse/YARN-11303 > Project: Hadoop YARN > Issue Type: Improvement > Components: security >Reporter: D M Murali Krishna Reddy >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > > The current jquery-ui version used(1.13.1) in the trunk has the following > vulnerability > [CVE-2022-31160|https://nvd.nist.gov/vuln/detail/CVE-2022-31160] so we need > to upgrade to at least 1.13.2. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11039) LogAggregationFileControllerFactory::getFileControllerForRead can leak threads
[
https://issues.apache.org/jira/browse/YARN-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-11039:
--
Summary: LogAggregationFileControllerFactory::getFileControllerForRead can
leak threads (was:
LogAggregationFileControllerFactory::getFileControllerForRead should close FS )
> LogAggregationFileControllerFactory::getFileControllerForRead can leak
> threads
> ---
>
> Key: YARN-11039
> URL: https://issues.apache.org/jira/browse/YARN-11039
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.3.4
>Reporter: Rajesh Balamohan
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: performance, stability
>
> getFileControllerForRead::getFileControllerForRead internally opens up a new
> FS object everytime and is not closed.
> When cloud connectors (e.g s3a) is used along with Knox, it ends up leaking
> KnoxTokenMonitor for every unclosed FS object causing thread leaks in NM.
> Lines of interest:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileControllerFactory.java#L167]
> {noformat}
>try {
> Path remoteAppLogDir = fileController.getOlderRemoteAppLogDir(appId,
> appOwner);
> if (LogAggregationUtils.getNodeFiles(conf, remoteAppLogDir, appId,
> appOwner).hasNext()) {
> return fileController;
> }
> } catch (Exception ex) {
> diagnosticsMsg.append(ex.getMessage() + "\n");
> continue;
> }
> {noformat}
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationUtils.java#L252]
> {noformat}
>
> public static RemoteIterator getNodeFiles(Configuration conf,
> Path remoteAppLogDir, ApplicationId appId, String appOwner)
> throws IOException {
> Path qualifiedLogDir =
> FileContext.getFileContext(conf).makeQualified(remoteAppLogDir);
> return FileContext.getFileContext(
> qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir);
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11039) LogAggregationFileControllerFactory::getFileControllerForRead should close FS
[
https://issues.apache.org/jira/browse/YARN-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reassigned YARN-11039:
-
Assignee: Steve Loughran
> LogAggregationFileControllerFactory::getFileControllerForRead should close FS
> --
>
> Key: YARN-11039
> URL: https://issues.apache.org/jira/browse/YARN-11039
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.3.4
>Reporter: Rajesh Balamohan
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: performance, stability
>
> getFileControllerForRead::getFileControllerForRead internally opens up a new
> FS object everytime and is not closed.
> When cloud connectors (e.g s3a) is used along with Knox, it ends up leaking
> KnoxTokenMonitor for every unclosed FS object causing thread leaks in NM.
> Lines of interest:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileControllerFactory.java#L167]
> {noformat}
>try {
> Path remoteAppLogDir = fileController.getOlderRemoteAppLogDir(appId,
> appOwner);
> if (LogAggregationUtils.getNodeFiles(conf, remoteAppLogDir, appId,
> appOwner).hasNext()) {
> return fileController;
> }
> } catch (Exception ex) {
> diagnosticsMsg.append(ex.getMessage() + "\n");
> continue;
> }
> {noformat}
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationUtils.java#L252]
> {noformat}
>
> public static RemoteIterator getNodeFiles(Configuration conf,
> Path remoteAppLogDir, ApplicationId appId, String appOwner)
> throws IOException {
> Path qualifiedLogDir =
> FileContext.getFileContext(conf).makeQualified(remoteAppLogDir);
> return FileContext.getFileContext(
> qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir);
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11039) LogAggregationFileControllerFactory::getFileControllerForRead should close FS
[
https://issues.apache.org/jira/browse/YARN-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612627#comment-17612627
]
Steve Loughran commented on YARN-11039:
---
problem here is that there is no FileContext.close(); the s3a and abfs adapters
need to have finalizers which close their wrapped fs instances.
I'd do it in DelegateToFS If i didn't think it was riskier. at least with the
abfs/s3a classes i know their close isn't reentrant
> LogAggregationFileControllerFactory::getFileControllerForRead should close FS
> --
>
> Key: YARN-11039
> URL: https://issues.apache.org/jira/browse/YARN-11039
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.3.4
>Reporter: Rajesh Balamohan
>Priority: Blocker
> Labels: performance, stability
>
> getFileControllerForRead::getFileControllerForRead internally opens up a new
> FS object everytime and is not closed.
> When cloud connectors (e.g s3a) is used along with Knox, it ends up leaking
> KnoxTokenMonitor for every unclosed FS object causing thread leaks in NM.
> Lines of interest:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileControllerFactory.java#L167]
> {noformat}
>try {
> Path remoteAppLogDir = fileController.getOlderRemoteAppLogDir(appId,
> appOwner);
> if (LogAggregationUtils.getNodeFiles(conf, remoteAppLogDir, appId,
> appOwner).hasNext()) {
> return fileController;
> }
> } catch (Exception ex) {
> diagnosticsMsg.append(ex.getMessage() + "\n");
> continue;
> }
> {noformat}
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationUtils.java#L252]
> {noformat}
>
> public static RemoteIterator getNodeFiles(Configuration conf,
> Path remoteAppLogDir, ApplicationId appId, String appOwner)
> throws IOException {
> Path qualifiedLogDir =
> FileContext.getFileContext(conf).makeQualified(remoteAppLogDir);
> return FileContext.getFileContext(
> qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir);
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11039) LogAggregationFileControllerFactory::getFileControllerForRead should close FS
[
https://issues.apache.org/jira/browse/YARN-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612623#comment-17612623
]
Steve Loughran commented on YARN-11039:
---
raising to a blocker as it can cause thread leakage if s3a/abfs is the dest
> LogAggregationFileControllerFactory::getFileControllerForRead should close FS
> --
>
> Key: YARN-11039
> URL: https://issues.apache.org/jira/browse/YARN-11039
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.3.4
>Reporter: Rajesh Balamohan
>Priority: Blocker
> Labels: performance, stability
>
> getFileControllerForRead::getFileControllerForRead internally opens up a new
> FS object everytime and is not closed.
> When cloud connectors (e.g s3a) is used along with Knox, it ends up leaking
> KnoxTokenMonitor for every unclosed FS object causing thread leaks in NM.
> Lines of interest:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileControllerFactory.java#L167]
> {noformat}
>try {
> Path remoteAppLogDir = fileController.getOlderRemoteAppLogDir(appId,
> appOwner);
> if (LogAggregationUtils.getNodeFiles(conf, remoteAppLogDir, appId,
> appOwner).hasNext()) {
> return fileController;
> }
> } catch (Exception ex) {
> diagnosticsMsg.append(ex.getMessage() + "\n");
> continue;
> }
> {noformat}
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationUtils.java#L252]
> {noformat}
>
> public static RemoteIterator getNodeFiles(Configuration conf,
> Path remoteAppLogDir, ApplicationId appId, String appOwner)
> throws IOException {
> Path qualifiedLogDir =
> FileContext.getFileContext(conf).makeQualified(remoteAppLogDir);
> return FileContext.getFileContext(
> qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir);
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11039) LogAggregationFileControllerFactory::getFileControllerForRead should close FS
[
https://issues.apache.org/jira/browse/YARN-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-11039:
--
Target Version/s: 3.3.5
> LogAggregationFileControllerFactory::getFileControllerForRead should close FS
> --
>
> Key: YARN-11039
> URL: https://issues.apache.org/jira/browse/YARN-11039
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.3.4
>Reporter: Rajesh Balamohan
>Priority: Blocker
> Labels: performance, stability
>
> getFileControllerForRead::getFileControllerForRead internally opens up a new
> FS object everytime and is not closed.
> When cloud connectors (e.g s3a) is used along with Knox, it ends up leaking
> KnoxTokenMonitor for every unclosed FS object causing thread leaks in NM.
> Lines of interest:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileControllerFactory.java#L167]
> {noformat}
>try {
> Path remoteAppLogDir = fileController.getOlderRemoteAppLogDir(appId,
> appOwner);
> if (LogAggregationUtils.getNodeFiles(conf, remoteAppLogDir, appId,
> appOwner).hasNext()) {
> return fileController;
> }
> } catch (Exception ex) {
> diagnosticsMsg.append(ex.getMessage() + "\n");
> continue;
> }
> {noformat}
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationUtils.java#L252]
> {noformat}
>
> public static RemoteIterator getNodeFiles(Configuration conf,
> Path remoteAppLogDir, ApplicationId appId, String appOwner)
> throws IOException {
> Path qualifiedLogDir =
> FileContext.getFileContext(conf).makeQualified(remoteAppLogDir);
> return FileContext.getFileContext(
> qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir);
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11039) LogAggregationFileControllerFactory::getFileControllerForRead should close FS
[
https://issues.apache.org/jira/browse/YARN-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-11039:
--
Priority: Blocker (was: Major)
> LogAggregationFileControllerFactory::getFileControllerForRead should close FS
> --
>
> Key: YARN-11039
> URL: https://issues.apache.org/jira/browse/YARN-11039
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: log-aggregation
>Affects Versions: 3.3.4
>Reporter: Rajesh Balamohan
>Priority: Blocker
> Labels: performance, stability
>
> getFileControllerForRead::getFileControllerForRead internally opens up a new
> FS object everytime and is not closed.
> When cloud connectors (e.g s3a) is used along with Knox, it ends up leaking
> KnoxTokenMonitor for every unclosed FS object causing thread leaks in NM.
> Lines of interest:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileControllerFactory.java#L167]
> {noformat}
>try {
> Path remoteAppLogDir = fileController.getOlderRemoteAppLogDir(appId,
> appOwner);
> if (LogAggregationUtils.getNodeFiles(conf, remoteAppLogDir, appId,
> appOwner).hasNext()) {
> return fileController;
> }
> } catch (Exception ex) {
> diagnosticsMsg.append(ex.getMessage() + "\n");
> continue;
> }
> {noformat}
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationUtils.java#L252]
> {noformat}
>
> public static RemoteIterator getNodeFiles(Configuration conf,
> Path remoteAppLogDir, ApplicationId appId, String appOwner)
> throws IOException {
> Path qualifiedLogDir =
> FileContext.getFileContext(conf).makeQualified(remoteAppLogDir);
> return FileContext.getFileContext(
> qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir);
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11039) LogAggregationFileControllerFactory::getFileControllerForRead should close FS
[
https://issues.apache.org/jira/browse/YARN-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-11039:
--
Issue Type: Bug (was: Improvement)
> LogAggregationFileControllerFactory::getFileControllerForRead should close FS
> --
>
> Key: YARN-11039
> URL: https://issues.apache.org/jira/browse/YARN-11039
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.3.4
>Reporter: Rajesh Balamohan
>Priority: Blocker
> Labels: performance, stability
>
> getFileControllerForRead::getFileControllerForRead internally opens up a new
> FS object everytime and is not closed.
> When cloud connectors (e.g s3a) is used along with Knox, it ends up leaking
> KnoxTokenMonitor for every unclosed FS object causing thread leaks in NM.
> Lines of interest:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileControllerFactory.java#L167]
> {noformat}
>try {
> Path remoteAppLogDir = fileController.getOlderRemoteAppLogDir(appId,
> appOwner);
> if (LogAggregationUtils.getNodeFiles(conf, remoteAppLogDir, appId,
> appOwner).hasNext()) {
> return fileController;
> }
> } catch (Exception ex) {
> diagnosticsMsg.append(ex.getMessage() + "\n");
> continue;
> }
> {noformat}
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationUtils.java#L252]
> {noformat}
>
> public static RemoteIterator getNodeFiles(Configuration conf,
> Path remoteAppLogDir, ApplicationId appId, String appOwner)
> throws IOException {
> Path qualifiedLogDir =
> FileContext.getFileContext(conf).makeQualified(remoteAppLogDir);
> return FileContext.getFileContext(
> qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir);
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11039) LogAggregationFileControllerFactory::getFileControllerForRead should close FS
[
https://issues.apache.org/jira/browse/YARN-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-11039:
--
Affects Version/s: 3.3.4
> LogAggregationFileControllerFactory::getFileControllerForRead should close FS
> --
>
> Key: YARN-11039
> URL: https://issues.apache.org/jira/browse/YARN-11039
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: log-aggregation
>Affects Versions: 3.3.4
>Reporter: Rajesh Balamohan
>Priority: Major
> Labels: performance, stability
>
> getFileControllerForRead::getFileControllerForRead internally opens up a new
> FS object everytime and is not closed.
> When cloud connectors (e.g s3a) is used along with Knox, it ends up leaking
> KnoxTokenMonitor for every unclosed FS object causing thread leaks in NM.
> Lines of interest:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileControllerFactory.java#L167]
> {noformat}
>try {
> Path remoteAppLogDir = fileController.getOlderRemoteAppLogDir(appId,
> appOwner);
> if (LogAggregationUtils.getNodeFiles(conf, remoteAppLogDir, appId,
> appOwner).hasNext()) {
> return fileController;
> }
> } catch (Exception ex) {
> diagnosticsMsg.append(ex.getMessage() + "\n");
> continue;
> }
> {noformat}
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationUtils.java#L252]
> {noformat}
>
> public static RemoteIterator getNodeFiles(Configuration conf,
> Path remoteAppLogDir, ApplicationId appId, String appOwner)
> throws IOException {
> Path qualifiedLogDir =
> FileContext.getFileContext(conf).makeQualified(remoteAppLogDir);
> return FileContext.getFileContext(
> qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir);
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11303) Upgrade jquery ui to 1.13.2
[ https://issues.apache.org/jira/browse/YARN-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608640#comment-17608640 ] Steve Loughran commented on YARN-11303: --- lets get this into branch-3.3 too > Upgrade jquery ui to 1.13.2 > --- > > Key: YARN-11303 > URL: https://issues.apache.org/jira/browse/YARN-11303 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: D M Murali Krishna Reddy >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The current jquery-ui version used(1.13.1) in the trunk has the following > vulnerability > [CVE-2022-31160|https://nvd.nist.gov/vuln/detail/CVE-2022-31160] so we need > to upgrade to at least 1.13.2. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11291) Can hadoop3.3.4 support spark job containing jars compiled on java17
[ https://issues.apache.org/jira/browse/YARN-11291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17599442#comment-17599442 ] Steve Loughran commented on YARN-11291: --- bq. I have installed a hadoop cluster versioned3.3.4 on java1.8. you might want to try upgrading the cluster. 3.3.4 has only been tested on java 11 though > Can hadoop3.3.4 support spark job containing jars compiled on java17 > > > Key: YARN-11291 > URL: https://issues.apache.org/jira/browse/YARN-11291 > Project: Hadoop YARN > Issue Type: Bug > Environment: hadoop: 3.3.4 > spark: 3.3.0 > java: 17 >Reporter: jiangjiguang0719 >Priority: Major > > I have installed a hadoop cluster versioned3.3.4 on java1.8. > Now I submit spark job to yarn, it does not works, and an error occurred > The spark job contains jars compiled on java17 . > How can I do to support spark job containing jar compiled on java17 ? > The error log is: > 22/09/01 13:18:09 WARN TaskSetManager: Lost task 0.0 in stage 2.0 (TID 1824) > (worker03 executor 34): java.lang.UnsupportedClassVersionError: > org/apache/parquet/column/values/bitpacking/Packer has been compiled by a > more recent version of the Java Runtime (class file version 61.0), this > version of the Java Runtime only recognizes class file versions up to 52.0 > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) > at java.net.URLClassLoader.access$100(URLClassLoader.java:74) > at java.net.URLClassLoader$1.run(URLClassLoader.java:369) > at java.net.URLClassLoader$1.run(URLClassLoader.java:363) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:362) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.init(VectorizedRleValuesReader.java:132) > at > org.apache.spark.sql.execution.datasources.parquet.VectorizedRleValuesReader.(VectorizedRleValuesReader.java:97) > at > org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readPageV2(VectorizedColumnReader.java:381) > at > org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.access$100(VectorizedColumnReader.java:49) > at > org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader$1.visit(VectorizedColumnReader.java:281) > at > org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader$1.visit(VectorizedColumnReader.java:268) > at > org.apache.parquet.column.page.DataPageV2.accept(DataPageV2.java:192) > at > org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readPage(VectorizedColumnReader.java:268) > at > org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:186) > at > org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:316) > at > org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:212) > at > org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39) > at > org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:116) > at > org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:274) > at > org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:116) > at > org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:553) > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.columnartorow_nextBatch_0$(Unknown > Source) > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown > Source) > at > org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) > at > org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760) > at > org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:364) > at >
[jira] [Updated] (YARN-11219) [Federation] Add getAppActivities, getAppStatistics REST APIs for Router
[ https://issues.apache.org/jira/browse/YARN-11219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11219: -- Fix Version/s: 3.3.9 (was: 3.3.4) > [Federation] Add getAppActivities, getAppStatistics REST APIs for Router > > > Key: YARN-11219 > URL: https://issues.apache.org/jira/browse/YARN-11219 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11193) hadoop-yarn-common jquery js file lacks approved header
[ https://issues.apache.org/jira/browse/YARN-11193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558757#comment-17558757 ] Steve Loughran commented on YARN-11193: --- bq. Also jquery and jquery-ui are separate upgrades the query ui patch, when cherrypicked, upated the exclusion list of the rat pom, though that may have been me getting the cherrypick merge conflict wrong. > hadoop-yarn-common jquery js file lacks approved header > --- > > Key: YARN-11193 > URL: https://issues.apache.org/jira/browse/YARN-11193 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.3.4 >Reporter: Steve Loughran >Priority: Blocker > > the updated jquery.js file gets rejected by rat. it either needs a valid > header or added to the exclusion list. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11193) hadoop-yarn-common jquery js file lacks approved header
[ https://issues.apache.org/jira/browse/YARN-11193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved YARN-11193. --- Resolution: Invalid > hadoop-yarn-common jquery js file lacks approved header > --- > > Key: YARN-11193 > URL: https://issues.apache.org/jira/browse/YARN-11193 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.3.4 >Reporter: Steve Loughran >Priority: Blocker > > the updated jquery.js file gets rejected by rat. it either needs a valid > header or added to the exclusion list. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11092) Upgrade jquery ui to 1.13.1
[ https://issues.apache.org/jira/browse/YARN-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11092: -- Fix Version/s: 3.3.4 (was: 3.3.9) > Upgrade jquery ui to 1.13.1 > --- > > Key: YARN-11092 > URL: https://issues.apache.org/jira/browse/YARN-11092 > Project: Hadoop YARN > Issue Type: Bug >Reporter: D M Murali Krishna Reddy >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.4 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The current jquery-ui version used(1.12.1) in the trunk has the following > vulnerabilities CVE-2021-41182, CVE-2021-41183, CVE-2021-41184, so we need to > upgrade to at least 1.13.0. > > Also currently for the UI2 we are using the shims repo which is not being > maintained as per the discussion > [https://github.com/components/jqueryui/issues/70] , so if possible we should > move to the main jquery repo [https://github.com/jquery/jquery-ui] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11092) Upgrade jquery ui to 1.13.1
[ https://issues.apache.org/jira/browse/YARN-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558052#comment-17558052 ] Steve Loughran commented on YARN-11092: --- this is more a drive-by regression than a deliberate one; the diff from this patch stamped on the jquery changes *adjacent* to it in the pom, updating them with a reference to the newer js file. i could have fixed the line there and in the .java file; went with pulling HADOOP-18044. still a bit of merge confilct, but upgrading was probably better than staying on the older release. at least i hope so > Upgrade jquery ui to 1.13.1 > --- > > Key: YARN-11092 > URL: https://issues.apache.org/jira/browse/YARN-11092 > Project: Hadoop YARN > Issue Type: Bug >Reporter: D M Murali Krishna Reddy >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.9 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The current jquery-ui version used(1.12.1) in the trunk has the following > vulnerabilities CVE-2021-41182, CVE-2021-41183, CVE-2021-41184, so we need to > upgrade to at least 1.13.0. > > Also currently for the UI2 we are using the shims repo which is not being > maintained as per the discussion > [https://github.com/components/jqueryui/issues/70] , so if possible we should > move to the main jquery repo [https://github.com/jquery/jquery-ui] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11193) hadoop-yarn-common jquery js file lacks approved header
[ https://issues.apache.org/jira/browse/YARN-11193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557604#comment-17557604 ] Steve Loughran commented on YARN-11193: --- yeah, just been looking at this myself; things are confusing. i will deal with tomorrow. this is a docker build and it should have been cleaned up completely before the test run, so maybe that cleanup isn't removing the old file. might be best to just checkout the yarn module again incidentally, looks like hdfs is on a different jquery version > hadoop-yarn-common jquery js file lacks approved header > --- > > Key: YARN-11193 > URL: https://issues.apache.org/jira/browse/YARN-11193 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.3.4 >Reporter: Steve Loughran >Priority: Blocker > > the updated jquery.js file gets rejected by rat. it either needs a valid > header or added to the exclusion list. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11193) hadoop-yarn-common jquery js file lacks approved header
[ https://issues.apache.org/jira/browse/YARN-11193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11193: -- Affects Version/s: (was: 3.4.0) (was: 3.3.9) > hadoop-yarn-common jquery js file lacks approved header > --- > > Key: YARN-11193 > URL: https://issues.apache.org/jira/browse/YARN-11193 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.3.4 >Reporter: Steve Loughran >Priority: Blocker > > the updated jquery.js file gets rejected by rat. it either needs a valid > header or added to the exclusion list. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11193) hadoop-yarn-common jquery js file lacks approved header
[ https://issues.apache.org/jira/browse/YARN-11193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11193: -- Summary: hadoop-yarn-common jquery js file lacks approved header (was: jquery js file lacks approved header) > hadoop-yarn-common jquery js file lacks approved header > --- > > Key: YARN-11193 > URL: https://issues.apache.org/jira/browse/YARN-11193 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.4.0, 3.3.9, 3.3.4 >Reporter: Steve Loughran >Priority: Blocker > > the updated jquery.js file gets rejected by rat. it either needs a valid > header or added to the exclusion list. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11193) jquery js file lacks approved header
[ https://issues.apache.org/jira/browse/YARN-11193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557500#comment-17557500 ] Steve Loughran commented on YARN-11193: --- doing some more checks to make sure this isn't just a tainted directory tree/mvn repo etc > jquery js file lacks approved header > > > Key: YARN-11193 > URL: https://issues.apache.org/jira/browse/YARN-11193 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.4.0, 3.3.9, 3.3.4 >Reporter: Steve Loughran >Priority: Blocker > > the updated jquery.js file gets rejected by rat. it either needs a valid > header or added to the exclusion list. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-11092) Upgrade jquery ui to 1.13.1
[
https://issues.apache.org/jira/browse/YARN-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557496#comment-17557496
]
Steve Loughran edited comment on YARN-11092 at 6/22/22 2:55 PM:
breaks the release in mvn apache-rat:check
{code}
> dev-support/bin/create-release --docker --dockercache
...
Apache RAT Check
$ /usr/bin/mvn -Dmaven.repo.local=/maven apache-rat:check >
/build/source/patchprocess/mvn_apache_rat.log 2>&1
Failed!
> find . -name rat.txt -print | xargs grep "unapproved"
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/rat.txt:Files with
unapproved licenses:
> cat ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/rat.txt
...
Printing headers for text files without a valid license header...
=
== File:
/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jqueary/jquery-3.5.1.min.js
{code}
was (Author: ste...@apache.org):
breaks the release in mvn apache-rat:check
{code}
> dev-support/bin/create-release --docker --dockercache
...
Apache RAT Check
$ /usr/bin/mvn -Dmaven.repo.local=/maven apache-rat:check >
/build/source/patchprocess/mvn_apache_rat.log 2>&1
Failed!
> cd patchprocess/
> find . -name rat.txt -print | xargs grep "unapproved"
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/rat.txt:Files with
unapproved licenses:
> cat ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/rat.txt
...
Printing headers for text files without a valid license header...
=
== File:
/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jquery/jquery-3.5.1.min.js
{code}
> Upgrade jquery ui to 1.13.1
> ---
>
> Key: YARN-11092
> URL: https://issues.apache.org/jira/browse/YARN-11092
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: D M Murali Krishna Reddy
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.9
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The current jquery-ui version used(1.12.1) in the trunk has the following
> vulnerabilities CVE-2021-41182, CVE-2021-41183, CVE-2021-41184, so we need to
> upgrade to at least 1.13.0.
>
> Also currently for the UI2 we are using the shims repo which is not being
> maintained as per the discussion
> [https://github.com/components/jqueryui/issues/70] , so if possible we should
> move to the main jquery repo [https://github.com/jquery/jquery-ui]
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11193) jquery js file lacks approved header
[
https://issues.apache.org/jira/browse/YARN-11193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557497#comment-17557497
]
Steve Loughran commented on YARN-11193:
---
breaks the release in mvn apache-rat:check
{code}
> dev-support/bin/create-release --docker --dockercache
...
Apache RAT Check
$ /usr/bin/mvn -Dmaven.repo.local=/maven apache-rat:check >
/build/source/patchprocess/mvn_apache_rat.log 2>&1
Failed!
> find . -name rat.txt -print | xargs grep "unapproved"
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/rat.txt:Files with
unapproved licenses:
> cat ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/rat.txt
...
Printing headers for text files without a valid license header...
=
== File:
/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jqueary/jquery-3.5.1.min.js
{code}
> jquery js file lacks approved header
>
>
> Key: YARN-11193
> URL: https://issues.apache.org/jira/browse/YARN-11193
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0, 3.3.9, 3.3.4
>Reporter: Steve Loughran
>Priority: Blocker
>
> the updated jquery.js file gets rejected by rat. it either needs a valid
> header or added to the exclusion list.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11193) jquery js file lacks approved header
Steve Loughran created YARN-11193: - Summary: jquery js file lacks approved header Key: YARN-11193 URL: https://issues.apache.org/jira/browse/YARN-11193 Project: Hadoop YARN Issue Type: Bug Components: build Affects Versions: 3.4.0, 3.3.9, 3.3.4 Reporter: Steve Loughran the updated jquery.js file gets rejected by rat. it either needs a valid header or added to the exclusion list. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11092) Upgrade jquery ui to 1.13.1
[
https://issues.apache.org/jira/browse/YARN-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557496#comment-17557496
]
Steve Loughran commented on YARN-11092:
---
breaks the release in mvn apache-rat:check
{code}
> dev-support/bin/create-release --docker --dockercache
...
Apache RAT Check
$ /usr/bin/mvn -Dmaven.repo.local=/maven apache-rat:check >
/build/source/patchprocess/mvn_apache_rat.log 2>&1
Failed!
> cd patchprocess/
> find . -name rat.txt -print | xargs grep "unapproved"
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/rat.txt:Files with
unapproved licenses:
> cat ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/rat.txt
...
Printing headers for text files without a valid license header...
=
== File:
/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jquery/jquery-3.5.1.min.js
{code}
> Upgrade jquery ui to 1.13.1
> ---
>
> Key: YARN-11092
> URL: https://issues.apache.org/jira/browse/YARN-11092
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: D M Murali Krishna Reddy
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.9
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The current jquery-ui version used(1.12.1) in the trunk has the following
> vulnerabilities CVE-2021-41182, CVE-2021-41183, CVE-2021-41184, so we need to
> upgrade to at least 1.13.0.
>
> Also currently for the UI2 we are using the shims repo which is not being
> maintained as per the discussion
> [https://github.com/components/jqueryui/issues/70] , so if possible we should
> move to the main jquery repo [https://github.com/jquery/jquery-ui]
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11189) TestLocalDistributedCacheManager failing after HADOOP-16202
[
https://issues.apache.org/jira/browse/YARN-11189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556416#comment-17556416
]
Steve Loughran commented on YARN-11189:
---
mockito test failure, probably caused by change in method invoked
{code}
java.io.IOException: java.util.concurrent.ExecutionException:
java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:140)
at
org.apache.hadoop.mapred.TestLocalDistributedCacheManager.testDownload(TestLocalDistributedCacheManager.java:186)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.util.concurrent.ExecutionException:
java.lang.reflect.UndeclaredThrowableException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:136)
... 31 more
Caused by: java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1935)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:422)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:71)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Download and unpack
failed
at
org.apache.hadoop.yarn.util.FSDownload.downloadAndUnpack(FSDownload.java:317)
at
org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:292)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:72)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:425)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:422)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1919)
... 6 more
[jira] [Created] (YARN-11189) TestLocalDistributedCacheManager failing after HADOOP-16202
Steve Loughran created YARN-11189: - Summary: TestLocalDistributedCacheManager failing after HADOOP-16202 Key: YARN-11189 URL: https://issues.apache.org/jira/browse/YARN-11189 Project: Hadoop YARN Issue Type: Bug Components: test Affects Versions: 3.4.0 Reporter: Steve Loughran Assignee: Steve Loughran After HADOOP-16202, TestLocalDistributedCacheManager.testDownload is failing with an NPE -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11172) Fix testDelegationToken
[
https://issues.apache.org/jira/browse/YARN-11172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved YARN-11172.
---
Fix Version/s: 3.4.0
3.3.4
Resolution: Fixed
> Fix testDelegationToken
> ---
>
> Key: YARN-11172
> URL: https://issues.apache.org/jira/browse/YARN-11172
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: test
>Affects Versions: 3.3.4
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.4
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> UT fail after HDFS-16563, other PR is blocked.
> {code:java}
> [ERROR]
> testDelegationToken(org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens)
> Time elapsed: 17.379 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.testDelegationToken(TestClientRMTokens.java:207)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11172) Fix testDelegationToken
[
https://issues.apache.org/jira/browse/YARN-11172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-11172:
--
Affects Version/s: 3.3.4
> Fix testDelegationToken
> ---
>
> Key: YARN-11172
> URL: https://issues.apache.org/jira/browse/YARN-11172
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 3.3.4
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> UT fail after HDFS-16563, other PR is blocked.
> {code:java}
> [ERROR]
> testDelegationToken(org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens)
> Time elapsed: 17.379 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.testDelegationToken(TestClientRMTokens.java:207)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11172) Fix testDelegationToken
[
https://issues.apache.org/jira/browse/YARN-11172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-11172:
--
Component/s: test
> Fix testDelegationToken
> ---
>
> Key: YARN-11172
> URL: https://issues.apache.org/jira/browse/YARN-11172
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: test
>Affects Versions: 3.3.4
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> UT fail after HDFS-16563, other PR is blocked.
> {code:java}
> [ERROR]
> testDelegationToken(org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens)
> Time elapsed: 17.379 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.testDelegationToken(TestClientRMTokens.java:207)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11173) update os-maven-plugin to 1.7.0 in yarn-csi
Steve Loughran created YARN-11173: - Summary: update os-maven-plugin to 1.7.0 in yarn-csi Key: YARN-11173 URL: https://issues.apache.org/jira/browse/YARN-11173 Project: Hadoop YARN Issue Type: Improvement Components: build Affects Versions: 3.3.4 Reporter: Steve Loughran Assignee: Steve Loughran update the yarn-csi import of os-maven-plugin to match HADOOP-18275 -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11014) YARN incorrectly validates maximum capacity resources on the validation API
[
https://issues.apache.org/jira/browse/YARN-11014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-11014:
--
Fix Version/s: 3.4.0
> YARN incorrectly validates maximum capacity resources on the validation API
> -
>
> Key: YARN-11014
> URL: https://issues.apache.org/jira/browse/YARN-11014
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.3
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> The YARN validation API
> (http://:8088/ws/v1/cluster/scheduler-conf/validate) allows a
> configuration in which a child queue has a greater maximum capacity value
> than its parent (the same update fails when it is attempted for real) using
> the DominantResourceCalculator.
> For example, the following passes validation when the root queue's maximum
> capacity is less than the 1 it is attempting to set here:
> {code:java}
> {
> "update-queue": [
> {
> "queue-name": "root.test",
> "params": {
> "entry": [
> {
> "key": "maximum-capacity",
> "value": "[memory=1,vcores=3]"
> },
> {
> "key": "capacity",
> "value": "[memory=3020,vcores=1]"
> }
> ]
> }
> }
> ]
> }
> {code}
> The reason for this is the newly created CapacityScheduler instance doesn't
> have the active nodes of the cluster in the nodeTracker object, hence the
> clusterResources will be Resources.none() during the init/reinit phase of
> this new instance.
> If the dominantRC has invalid denominator (clusterResource being 0) it simply
> compares the two values resource-by-resource:
> {code:java}
> @Override
> public int compare(Resource clusterResource, Resource lhs, Resource rhs,
> boolean singleType) {
> if (lhs.equals(rhs)) {
> return 0;
> }
> if (isAllInvalidDivisor(clusterResource)) {
> return this.compare(lhs, rhs);
> }
> ...
> /**
>* Compare two resources - if the value for every resource type for the lhs
>* is greater than that of the rhs, return 1. If the value for every
> resource
>* type in the lhs is less than the rhs, return -1. Otherwise, return 0
>*
>* @param lhs resource to be compared
>* @param rhs resource to be compared
>* @return 0, 1, or -1
>*/
>private int compare(Resource lhs, Resource rhs) {
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11075) Explicitly declare serialVersionUID in LogMutation class
[ https://issues.apache.org/jira/browse/YARN-11075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11075: -- Fix Version/s: 3.4.0 > Explicitly declare serialVersionUID in LogMutation class > > > Key: YARN-11075 > URL: https://issues.apache.org/jira/browse/YARN-11075 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.3.1 >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.1.5, 3.3.3 > > Time Spent: 4h > Remaining Estimate: 0h > > The > [LogMutation|https://github.com/apache/hadoop/blob/f91e21ac109e753e76d19c5c872c59a767b7b837/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/YarnConfigurationStore.java#L55] > class doesn't have its serialVersionUID defined as a private final static > constant hence the compiler generates one for it. Upon every > [logMutation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java#L163] > call, the existing logMutation objects are collected from the chosen config > store and deserialized, but if a new/different RM jar loads these and the > serialVersionUIDs don't match (for example a different compiler generated a > different ID) an InvalidClassException occurs. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11075) Explicitly declare serialVersionUID in LogMutation class
[ https://issues.apache.org/jira/browse/YARN-11075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved YARN-11075. --- Fix Version/s: 3.3.3 (was: 3.4.0) (was: 3.3.4) Resolution: Fixed fFIxed in 3.3.3; updating fix versions as appropriate > Explicitly declare serialVersionUID in LogMutation class > > > Key: YARN-11075 > URL: https://issues.apache.org/jira/browse/YARN-11075 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.3.1 >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Labels: pull-request-available > Fix For: 3.2.4, 3.1.5, 3.3.3 > > Time Spent: 4h > Remaining Estimate: 0h > > The > [LogMutation|https://github.com/apache/hadoop/blob/f91e21ac109e753e76d19c5c872c59a767b7b837/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/YarnConfigurationStore.java#L55] > class doesn't have its serialVersionUID defined as a private final static > constant hence the compiler generates one for it. Upon every > [logMutation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java#L163] > call, the existing logMutation objects are collected from the chosen config > store and deserialized, but if a new/different RM jar loads these and the > serialVersionUIDs don't match (for example a different compiler generated a > different ID) an InvalidClassException occurs. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11014) YARN incorrectly validates maximum capacity resources on the validation API
[
https://issues.apache.org/jira/browse/YARN-11014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved YARN-11014.
---
Fix Version/s: 3.3.3
(was: 3.4.0)
(was: 3.3.4)
Resolution: Fixed
> YARN incorrectly validates maximum capacity resources on the validation API
> -
>
> Key: YARN-11014
> URL: https://issues.apache.org/jira/browse/YARN-11014
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.4, 3.3.3
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> The YARN validation API
> (http://:8088/ws/v1/cluster/scheduler-conf/validate) allows a
> configuration in which a child queue has a greater maximum capacity value
> than its parent (the same update fails when it is attempted for real) using
> the DominantResourceCalculator.
> For example, the following passes validation when the root queue's maximum
> capacity is less than the 1 it is attempting to set here:
> {code:java}
> {
> "update-queue": [
> {
> "queue-name": "root.test",
> "params": {
> "entry": [
> {
> "key": "maximum-capacity",
> "value": "[memory=1,vcores=3]"
> },
> {
> "key": "capacity",
> "value": "[memory=3020,vcores=1]"
> }
> ]
> }
> }
> ]
> }
> {code}
> The reason for this is the newly created CapacityScheduler instance doesn't
> have the active nodes of the cluster in the nodeTracker object, hence the
> clusterResources will be Resources.none() during the init/reinit phase of
> this new instance.
> If the dominantRC has invalid denominator (clusterResource being 0) it simply
> compares the two values resource-by-resource:
> {code:java}
> @Override
> public int compare(Resource clusterResource, Resource lhs, Resource rhs,
> boolean singleType) {
> if (lhs.equals(rhs)) {
> return 0;
> }
> if (isAllInvalidDivisor(clusterResource)) {
> return this.compare(lhs, rhs);
> }
> ...
> /**
>* Compare two resources - if the value for every resource type for the lhs
>* is greater than that of the rhs, return 1. If the value for every
> resource
>* type in the lhs is less than the rhs, return -1. Otherwise, return 0
>*
>* @param lhs resource to be compared
>* @param rhs resource to be compared
>* @return 0, 1, or -1
>*/
>private int compare(Resource lhs, Resource rhs) {
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11014) YARN incorrectly validates maximum capacity resources on the validation API
[
https://issues.apache.org/jira/browse/YARN-11014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17523719#comment-17523719
]
Steve Loughran commented on YARN-11014:
---
FIxed in 3.3.3; updating fix versions as appropriate
> YARN incorrectly validates maximum capacity resources on the validation API
> -
>
> Key: YARN-11014
> URL: https://issues.apache.org/jira/browse/YARN-11014
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.4, 3.3.3
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> The YARN validation API
> (http://:8088/ws/v1/cluster/scheduler-conf/validate) allows a
> configuration in which a child queue has a greater maximum capacity value
> than its parent (the same update fails when it is attempted for real) using
> the DominantResourceCalculator.
> For example, the following passes validation when the root queue's maximum
> capacity is less than the 1 it is attempting to set here:
> {code:java}
> {
> "update-queue": [
> {
> "queue-name": "root.test",
> "params": {
> "entry": [
> {
> "key": "maximum-capacity",
> "value": "[memory=1,vcores=3]"
> },
> {
> "key": "capacity",
> "value": "[memory=3020,vcores=1]"
> }
> ]
> }
> }
> ]
> }
> {code}
> The reason for this is the newly created CapacityScheduler instance doesn't
> have the active nodes of the cluster in the nodeTracker object, hence the
> clusterResources will be Resources.none() during the init/reinit phase of
> this new instance.
> If the dominantRC has invalid denominator (clusterResource being 0) it simply
> compares the two values resource-by-resource:
> {code:java}
> @Override
> public int compare(Resource clusterResource, Resource lhs, Resource rhs,
> boolean singleType) {
> if (lhs.equals(rhs)) {
> return 0;
> }
> if (isAllInvalidDivisor(clusterResource)) {
> return this.compare(lhs, rhs);
> }
> ...
> /**
>* Compare two resources - if the value for every resource type for the lhs
>* is greater than that of the rhs, return 1. If the value for every
> resource
>* type in the lhs is less than the rhs, return -1. Otherwise, return 0
>*
>* @param lhs resource to be compared
>* @param rhs resource to be compared
>* @return 0, 1, or -1
>*/
>private int compare(Resource lhs, Resource rhs) {
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10720) YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging
[
https://issues.apache.org/jira/browse/YARN-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved YARN-10720.
---
Fix Version/s: 3.3.3
Resolution: Fixed
FIxed in 3.3.3; updating fix versions as appropriate
> YARN WebAppProxyServlet should support connection timeout to prevent proxy
> server from hanging
> --
>
> Key: YARN-10720
> URL: https://issues.apache.org/jira/browse/YARN-10720
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 2.10.2, 3.2.4, 3.3.3
>
> Attachments: YARN-10720.001.patch, YARN-10720.002.patch,
> YARN-10720.003.patch, YARN-10720.004.patch, YARN-10720.005.patch,
> YARN-10720.006.patch, image-2021-03-29-14-04-33-776.png,
> image-2021-03-29-14-05-32-708.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Following is proxy server show, {color:#de350b}too many connections from one
> client{color}, this caused the proxy server hang, and the yarn web can't jump
> to web proxy.
> !image-2021-03-29-14-04-33-776.png|width=632,height=57!
> Following is the AM which is abnormal, but proxy server don't know it is
> abnormal already, so the connections can't be closed, we should add time out
> support in proxy server to prevent this. And one abnormal AM may cause
> hundreds even thousands of connections, it is very heavy.
> !image-2021-03-29-14-05-32-708.png|width=669,height=101!
>
> After i kill the abnormal AM, the proxy server become healthy. This case
> happened many times in our production clusters, our clusters are huge, and
> the abnormal AM will be existed in a regular case.
>
> I will add timeout supported in web proxy server in this jira.
>
> cc [~pbacsko] [~ebadger] [~Jim_Brennan] [~ztang] [~epayne] [~gandras]
> [~bteke]
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10720) YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging
[
https://issues.apache.org/jira/browse/YARN-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-10720:
--
Fix Version/s: (was: 3.4.0)
(was: 3.3.4)
> YARN WebAppProxyServlet should support connection timeout to prevent proxy
> server from hanging
> --
>
> Key: YARN-10720
> URL: https://issues.apache.org/jira/browse/YARN-10720
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 2.10.2, 3.2.4
>
> Attachments: YARN-10720.001.patch, YARN-10720.002.patch,
> YARN-10720.003.patch, YARN-10720.004.patch, YARN-10720.005.patch,
> YARN-10720.006.patch, image-2021-03-29-14-04-33-776.png,
> image-2021-03-29-14-05-32-708.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Following is proxy server show, {color:#de350b}too many connections from one
> client{color}, this caused the proxy server hang, and the yarn web can't jump
> to web proxy.
> !image-2021-03-29-14-04-33-776.png|width=632,height=57!
> Following is the AM which is abnormal, but proxy server don't know it is
> abnormal already, so the connections can't be closed, we should add time out
> support in proxy server to prevent this. And one abnormal AM may cause
> hundreds even thousands of connections, it is very heavy.
> !image-2021-03-29-14-05-32-708.png|width=669,height=101!
>
> After i kill the abnormal AM, the proxy server become healthy. This case
> happened many times in our production clusters, our clusters are huge, and
> the abnormal AM will be existed in a regular case.
>
> I will add timeout supported in web proxy server in this jira.
>
> cc [~pbacsko] [~ebadger] [~Jim_Brennan] [~ztang] [~epayne] [~gandras]
> [~bteke]
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11075) Explicitly declare serialVersionUID in LogMutation class
[ https://issues.apache.org/jira/browse/YARN-11075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11075: -- Target Version/s: 3.3.3 > Explicitly declare serialVersionUID in LogMutation class > > > Key: YARN-11075 > URL: https://issues.apache.org/jira/browse/YARN-11075 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.3.1 >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.1.5, 3.3.4 > > Time Spent: 4h > Remaining Estimate: 0h > > The > [LogMutation|https://github.com/apache/hadoop/blob/f91e21ac109e753e76d19c5c872c59a767b7b837/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/YarnConfigurationStore.java#L55] > class doesn't have its serialVersionUID defined as a private final static > constant hence the compiler generates one for it. Upon every > [logMutation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java#L163] > call, the existing logMutation objects are collected from the chosen config > store and deserialized, but if a new/different RM jar loads these and the > serialVersionUIDs don't match (for example a different compiler generated a > different ID) an InvalidClassException occurs. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11014) YARN incorrectly validates maximum capacity resources on the validation API
[
https://issues.apache.org/jira/browse/YARN-11014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-11014:
--
Target Version/s: 3.3.3
> YARN incorrectly validates maximum capacity resources on the validation API
> -
>
> Key: YARN-11014
> URL: https://issues.apache.org/jira/browse/YARN-11014
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> The YARN validation API
> (http://:8088/ws/v1/cluster/scheduler-conf/validate) allows a
> configuration in which a child queue has a greater maximum capacity value
> than its parent (the same update fails when it is attempted for real) using
> the DominantResourceCalculator.
> For example, the following passes validation when the root queue's maximum
> capacity is less than the 1 it is attempting to set here:
> {code:java}
> {
> "update-queue": [
> {
> "queue-name": "root.test",
> "params": {
> "entry": [
> {
> "key": "maximum-capacity",
> "value": "[memory=1,vcores=3]"
> },
> {
> "key": "capacity",
> "value": "[memory=3020,vcores=1]"
> }
> ]
> }
> }
> ]
> }
> {code}
> The reason for this is the newly created CapacityScheduler instance doesn't
> have the active nodes of the cluster in the nodeTracker object, hence the
> clusterResources will be Resources.none() during the init/reinit phase of
> this new instance.
> If the dominantRC has invalid denominator (clusterResource being 0) it simply
> compares the two values resource-by-resource:
> {code:java}
> @Override
> public int compare(Resource clusterResource, Resource lhs, Resource rhs,
> boolean singleType) {
> if (lhs.equals(rhs)) {
> return 0;
> }
> if (isAllInvalidDivisor(clusterResource)) {
> return this.compare(lhs, rhs);
> }
> ...
> /**
>* Compare two resources - if the value for every resource type for the lhs
>* is greater than that of the rhs, return 1. If the value for every
> resource
>* type in the lhs is less than the rhs, return -1. Otherwise, return 0
>*
>* @param lhs resource to be compared
>* @param rhs resource to be compared
>* @return 0, 1, or -1
>*/
>private int compare(Resource lhs, Resource rhs) {
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-11075) Explicitly declare serialVersionUID in LogMutation class
[ https://issues.apache.org/jira/browse/YARN-11075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reopened YARN-11075: --- > Explicitly declare serialVersionUID in LogMutation class > > > Key: YARN-11075 > URL: https://issues.apache.org/jira/browse/YARN-11075 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.3.1 >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.1.5, 3.3.4 > > Time Spent: 4h > Remaining Estimate: 0h > > The > [LogMutation|https://github.com/apache/hadoop/blob/f91e21ac109e753e76d19c5c872c59a767b7b837/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/YarnConfigurationStore.java#L55] > class doesn't have its serialVersionUID defined as a private final static > constant hence the compiler generates one for it. Upon every > [logMutation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java#L163] > call, the existing logMutation objects are collected from the chosen config > store and deserialized, but if a new/different RM jar loads these and the > serialVersionUIDs don't match (for example a different compiler generated a > different ID) an InvalidClassException occurs. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-11014) YARN incorrectly validates maximum capacity resources on the validation API
[
https://issues.apache.org/jira/browse/YARN-11014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reopened YARN-11014:
---
> YARN incorrectly validates maximum capacity resources on the validation API
> -
>
> Key: YARN-11014
> URL: https://issues.apache.org/jira/browse/YARN-11014
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> The YARN validation API
> (http://:8088/ws/v1/cluster/scheduler-conf/validate) allows a
> configuration in which a child queue has a greater maximum capacity value
> than its parent (the same update fails when it is attempted for real) using
> the DominantResourceCalculator.
> For example, the following passes validation when the root queue's maximum
> capacity is less than the 1 it is attempting to set here:
> {code:java}
> {
> "update-queue": [
> {
> "queue-name": "root.test",
> "params": {
> "entry": [
> {
> "key": "maximum-capacity",
> "value": "[memory=1,vcores=3]"
> },
> {
> "key": "capacity",
> "value": "[memory=3020,vcores=1]"
> }
> ]
> }
> }
> ]
> }
> {code}
> The reason for this is the newly created CapacityScheduler instance doesn't
> have the active nodes of the cluster in the nodeTracker object, hence the
> clusterResources will be Resources.none() during the init/reinit phase of
> this new instance.
> If the dominantRC has invalid denominator (clusterResource being 0) it simply
> compares the two values resource-by-resource:
> {code:java}
> @Override
> public int compare(Resource clusterResource, Resource lhs, Resource rhs,
> boolean singleType) {
> if (lhs.equals(rhs)) {
> return 0;
> }
> if (isAllInvalidDivisor(clusterResource)) {
> return this.compare(lhs, rhs);
> }
> ...
> /**
>* Compare two resources - if the value for every resource type for the lhs
>* is greater than that of the rhs, return 1. If the value for every
> resource
>* type in the lhs is less than the rhs, return -1. Otherwise, return 0
>*
>* @param lhs resource to be compared
>* @param rhs resource to be compared
>* @return 0, 1, or -1
>*/
>private int compare(Resource lhs, Resource rhs) {
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10720) YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging
[
https://issues.apache.org/jira/browse/YARN-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-10720:
--
Target Version/s: 3.3.3
> YARN WebAppProxyServlet should support connection timeout to prevent proxy
> server from hanging
> --
>
> Key: YARN-10720
> URL: https://issues.apache.org/jira/browse/YARN-10720
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.2.4, 3.3.4
>
> Attachments: YARN-10720.001.patch, YARN-10720.002.patch,
> YARN-10720.003.patch, YARN-10720.004.patch, YARN-10720.005.patch,
> YARN-10720.006.patch, image-2021-03-29-14-04-33-776.png,
> image-2021-03-29-14-05-32-708.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Following is proxy server show, {color:#de350b}too many connections from one
> client{color}, this caused the proxy server hang, and the yarn web can't jump
> to web proxy.
> !image-2021-03-29-14-04-33-776.png|width=632,height=57!
> Following is the AM which is abnormal, but proxy server don't know it is
> abnormal already, so the connections can't be closed, we should add time out
> support in proxy server to prevent this. And one abnormal AM may cause
> hundreds even thousands of connections, it is very heavy.
> !image-2021-03-29-14-05-32-708.png|width=669,height=101!
>
> After i kill the abnormal AM, the proxy server become healthy. This case
> happened many times in our production clusters, our clusters are huge, and
> the abnormal AM will be existed in a regular case.
>
> I will add timeout supported in web proxy server in this jira.
>
> cc [~pbacsko] [~ebadger] [~Jim_Brennan] [~ztang] [~epayne] [~gandras]
> [~bteke]
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-10720) YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging
[
https://issues.apache.org/jira/browse/YARN-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reopened YARN-10720:
---
> YARN WebAppProxyServlet should support connection timeout to prevent proxy
> server from hanging
> --
>
> Key: YARN-10720
> URL: https://issues.apache.org/jira/browse/YARN-10720
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.2.4, 3.3.4
>
> Attachments: YARN-10720.001.patch, YARN-10720.002.patch,
> YARN-10720.003.patch, YARN-10720.004.patch, YARN-10720.005.patch,
> YARN-10720.006.patch, image-2021-03-29-14-04-33-776.png,
> image-2021-03-29-14-05-32-708.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Following is proxy server show, {color:#de350b}too many connections from one
> client{color}, this caused the proxy server hang, and the yarn web can't jump
> to web proxy.
> !image-2021-03-29-14-04-33-776.png|width=632,height=57!
> Following is the AM which is abnormal, but proxy server don't know it is
> abnormal already, so the connections can't be closed, we should add time out
> support in proxy server to prevent this. And one abnormal AM may cause
> hundreds even thousands of connections, it is very heavy.
> !image-2021-03-29-14-05-32-708.png|width=669,height=101!
>
> After i kill the abnormal AM, the proxy server become healthy. This case
> happened many times in our production clusters, our clusters are huge, and
> the abnormal AM will be existed in a regular case.
>
> I will add timeout supported in web proxy server in this jira.
>
> cc [~pbacsko] [~ebadger] [~Jim_Brennan] [~ztang] [~epayne] [~gandras]
> [~bteke]
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11085) Migrate to use latest and secured log4j2 version from log4j
[ https://issues.apache.org/jira/browse/YARN-11085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-11085: -- Component/s: build > Migrate to use latest and secured log4j2 version from log4j > --- > > Key: YARN-11085 > URL: https://issues.apache.org/jira/browse/YARN-11085 > Project: Hadoop YARN > Issue Type: Improvement > Components: build >Reporter: HIMANSHU GARG >Priority: Minor > > Migrate to use latest and secured log4j2 version from log4j > [https://github.com/apache/hadoop/blob/66b72406bd8bed28ca32c75e07fc2b682500e92b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml#L69] > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10561) Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN application catalog webapp
[ https://issues.apache.org/jira/browse/YARN-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17483709#comment-17483709 ] Steve Loughran commented on YARN-10561: --- thanks for fixing this! > Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN application catalog > webapp > > > Key: YARN-10561 > URL: https://issues.apache.org/jira/browse/YARN-10561 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0, 3.3.3 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > YARN application catalog webapp is using node.js 8.11.3, and 8.x are already > EoL. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11013) Provide a public wrapper of Configuration#substituteVars
[ https://issues.apache.org/jira/browse/YARN-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447941#comment-17447941 ] Steve Loughran commented on YARN-11013: --- # moving to HADOOP jira for visibility there; this is more than just a YARN issue # which versions has the original change gone in? Can you mark that? > Provide a public wrapper of Configuration#substituteVars > > > Key: YARN-11013 > URL: https://issues.apache.org/jira/browse/YARN-11013 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > YARN-10838 and YARN-10911 introduced an unfortunate change in Configuration > that could potentially be backward incompatible (visibility of > substituteVars). Since it is easily circumvented, my proposal is to avoid > breaking changes if possible. > One found issue is that Oozie defines substituteVars as a private method. > https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/util/XConfiguration.java#L186 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10992) schema generation produces code which doesn't compile on java 11+
[ https://issues.apache.org/jira/browse/YARN-10992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436093#comment-17436093 ] Steve Loughran commented on YARN-10992: --- think it may be the other way round -code generated in a later vm won't compile with the older ones > schema generation produces code which doesn't compile on java 11+ > - > > Key: YARN-10992 > URL: https://issues.apache.org/jira/browse/YARN-10992 > Project: Hadoop YARN > Issue Type: Sub-task > Components: buid >Affects Versions: 3.4.0 > Environment: build with openjdk 17 >Reporter: Steve Loughran >Priority: Major > > The changes of YARN-10386 are somehow stopping java 17 (maybe 11_+?) > compilation, as the generated classes have an import > javax.annotation.Generated which is not in the JRE classes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17435352#comment-17435352 ] Steve Loughran commented on YARN-10386: --- done: YARN-10992 > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.4.0 > > Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch, > YARN-10386-002.patch, YARN-10386-003.patch, YARN-10386-004.patch, > YARN-10386-005.patch, YARN-10386-006.patch, YARN-10386-007.patch, > YARN-10386-008.patch, YARN-10386-appendum.patch, YARN-10386-appendum2.patch > > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10992) schema generation produces code which doesn't compile on java 11+
[
https://issues.apache.org/jira/browse/YARN-10992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17435351#comment-17435351
]
Steve Loughran commented on YARN-10992:
---
build failure
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-yarn-server-resourcemanager: Compilation failure: Compilation
failure:
[ERROR]
/Users/stevel/hadoop/commit/apache-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/schema/MappingRulesDescription.java:[8,24]
cannot find symbol
[ERROR] symbol: class Generated
[ERROR] location: package javax.annotation
[ERROR]
/Users/stevel/hadoop/commit/apache-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/schema/MappingRulesDescription.java:[20,2]
cannot find symbol
[ERROR] symbol: class Generated
[ERROR]
/Users/stevel/hadoop/commit/apache-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/schema/Rule.java:[6,24]
cannot find symbol
[ERROR] symbol: class Generated
[ERROR] location: package javax.annotation
[ERROR]
/Users/stevel/hadoop/commit/apache-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/schema/Rule.java:[27,2]
cannot find symbol
[ERROR] symbol: class Generated
[ERROR]
/Users/stevel/hadoop/commit/apache-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/schema/Rule.java:[304,6]
cannot find symbol
[ERROR] symbol: class Generated
[ERROR] location: class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.schema.Rule
[ERROR]
/Users/stevel/hadoop/commit/apache-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/schema/Rule.java:[255,6]
cannot find symbol
[ERROR] symbol: class Generated
[ERROR] location: class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.schema.Rule
[ERROR]
/Users/stevel/hadoop/commit/apache-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/schema/Rule.java:[214,6]
cannot find symbol
[ERROR] symbol: class Generated
[ERROR] location: class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.schema.Rule
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-yarn-server-resourcemanager
{code}
> schema generation produces code which doesn't compile on java 11+
> -
>
> Key: YARN-10992
> URL: https://issues.apache.org/jira/browse/YARN-10992
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: buid
>Affects Versions: 3.4.0
> Environment: build with openjdk 17
>Reporter: Steve Loughran
>Priority: Major
>
> The changes of YARN-10386 are somehow stopping java 17 (maybe 11_+?)
> compilation, as the generated classes have an import
> javax.annotation.Generated which is not in the JRE classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10992) schema generation produces code which doesn't compile on java 11+
Steve Loughran created YARN-10992: - Summary: schema generation produces code which doesn't compile on java 11+ Key: YARN-10992 URL: https://issues.apache.org/jira/browse/YARN-10992 Project: Hadoop YARN Issue Type: Sub-task Components: buid Affects Versions: 3.4.0 Environment: build with openjdk 17 Reporter: Steve Loughran The changes of YARN-10386 are somehow stopping java 17 (maybe 11_+?) compilation, as the generated classes have an import javax.annotation.Generated which is not in the JRE classes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10988) Spark application stuck at ACCEPTED state at spark-submit
[ https://issues.apache.org/jira/browse/YARN-10988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17433860#comment-17433860 ] Steve Loughran commented on YARN-10988: --- https://spark.apache.org/docs/latest/configuration.html look @ core and memory requirements; ask for smaller values > Spark application stuck at ACCEPTED state at spark-submit > - > > Key: YARN-10988 > URL: https://issues.apache.org/jira/browse/YARN-10988 > Project: Hadoop YARN > Issue Type: Bug > Components: applications >Affects Versions: 3.2.1 >Reporter: unical1988 >Priority: Major > > Hello, > > I have configured & set Hadoop Cluster over 2 nodes and launch it along with > Yarn like so : > > *_On the master node :_* > * hdfs namenode -regular > * yarn resourcemanager > > *_On the slave node :_* > * hdfs datanode -regular > * yarn nodemanager > And it shows through UI that there has been a connection established between > the two machines that form the cluster. > To note that *_start-dfs_* on the master node started both namenode and > datanode even after setting *_slaves_* and *_hosts_* files. > Now i submit an application (simple _hello world_) to _Yarn_ : through this > command : > *Spark-submit --class "main" --master yarn pathToJar* > > But i get the error > 15/08/29 12:07:58 INFO Client: ApplicationManager is waiting for the > ResourceManager > client token: N/A diagnostics: N/A > ApplicationMaster host: N/A > ApplicationMaster RPC port: -1 > queue: root.hdfs > start time: 1440864477580 > final status: UNDEFINED tracking URL: > http://chd2.moneyball.guru:8088/proxy/application_1440861466017_0007/ user: > hdfs 15/08/29 12:07:59 INFO Client: Application report for > application_1440861466017_0007 (state: ACCEPTED) 15/08/29 12:08:00 INFO > Client: Application report for application_1440861466017_0007 (state: > ACCEPTED) 15/08/29 12:08:01 INFO Client: Application report for > application_1440861466017_0007 (state: ACCEPTED)... > > What am i missing in my configuration ? > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10988) Spark application stuck at ACCEPTED state at spark-submit
[ https://issues.apache.org/jira/browse/YARN-10988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17433097#comment-17433097 ] Steve Loughran commented on YARN-10988: --- * jIRA is for bug reports, not support issues. sorry * you probably don't have enough capacity for the #of cores your job wants. > Spark application stuck at ACCEPTED state at spark-submit > - > > Key: YARN-10988 > URL: https://issues.apache.org/jira/browse/YARN-10988 > Project: Hadoop YARN > Issue Type: Bug > Components: applications >Affects Versions: 3.2.1 >Reporter: unical1988 >Priority: Major > > Hello, > > I have configured & set Hadoop Cluster over 2 nodes and launch it along with > Yarn like so : > > *_On the master node :_* > * hdfs namenode -regular > * yarn resourcemanager > > *_On the slave node :_* > * hdfs datanode -regular > * yarn nodemanager > And it shows through UI that there has been a connection established between > the two machines that form the cluster. > To note that *_start-dfs_* on the master node started both namenode and > datanode even after setting *_slaves_* and *_hosts_* files. > Now i submit an application (simple _hello world_) to _Yarn_ : through this > command : > *Spark-submit --class "main" --master yarn pathToJar* > > But i get the error > 15/08/29 12:07:58 INFO Client: ApplicationManager is waiting for the > ResourceManager > client token: N/A diagnostics: N/A > ApplicationMaster host: N/A > ApplicationMaster RPC port: -1 > queue: root.hdfs > start time: 1440864477580 > final status: UNDEFINED tracking URL: > http://chd2.moneyball.guru:8088/proxy/application_1440861466017_0007/ user: > hdfs 15/08/29 12:07:59 INFO Client: Application report for > application_1440861466017_0007 (state: ACCEPTED) 15/08/29 12:08:00 INFO > Client: Application report for application_1440861466017_0007 (state: > ACCEPTED) 15/08/29 12:08:01 INFO Client: Application report for > application_1440861466017_0007 (state: ACCEPTED)... > > What am i missing in my configuration ? > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10988) Spark application stuck at ACCEPTED state at spark-submit
[ https://issues.apache.org/jira/browse/YARN-10988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved YARN-10988. --- Resolution: Invalid > Spark application stuck at ACCEPTED state at spark-submit > - > > Key: YARN-10988 > URL: https://issues.apache.org/jira/browse/YARN-10988 > Project: Hadoop YARN > Issue Type: Bug > Components: applications >Affects Versions: 3.2.1 >Reporter: unical1988 >Priority: Major > > Hello, > > I have configured & set Hadoop Cluster over 2 nodes and launch it along with > Yarn like so : > > *_On the master node :_* > * hdfs namenode -regular > * yarn resourcemanager > > *_On the slave node :_* > * hdfs datanode -regular > * yarn nodemanager > And it shows through UI that there has been a connection established between > the two machines that form the cluster. > To note that *_start-dfs_* on the master node started both namenode and > datanode even after setting *_slaves_* and *_hosts_* files. > Now i submit an application (simple _hello world_) to _Yarn_ : through this > command : > *Spark-submit --class "main" --master yarn pathToJar* > > But i get the error > 15/08/29 12:07:58 INFO Client: ApplicationManager is waiting for the > ResourceManager > client token: N/A diagnostics: N/A > ApplicationMaster host: N/A > ApplicationMaster RPC port: -1 > queue: root.hdfs > start time: 1440864477580 > final status: UNDEFINED tracking URL: > http://chd2.moneyball.guru:8088/proxy/application_1440861466017_0007/ user: > hdfs 15/08/29 12:07:59 INFO Client: Application report for > application_1440861466017_0007 (state: ACCEPTED) 15/08/29 12:08:00 INFO > Client: Application report for application_1440861466017_0007 (state: > ACCEPTED) 15/08/29 12:08:01 INFO Client: Application report for > application_1440861466017_0007 (state: ACCEPTED)... > > What am i missing in my configuration ? > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10386) Create new JSON schema for Placement Rules
[
https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17432649#comment-17432649
]
Steve Loughran commented on YARN-10386:
---
This is somehow stopping java 17 (maybe 11_+?) compilation, as the generated
classes have an import javax.annotation.Generated which is not in the JRE
classes.
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-yarn-server-resourcemanager: Compilation failure: Compilation
failure:
[ERROR]
/Users/stevel/hadoop/commit/apache-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/schema/MappingRulesDescription.java:[8,24]
cannot find symbol
[ERROR] symbol: class Generated
[ERROR] location: package javax.annotation
[ERROR]
/Users/stevel/hadoop/commit/apache-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/schema/MappingRulesDescription.java:[20,2]
cannot find symbol
[ERROR] symbol: class Generated
[ERROR]
/Users/stevel/hadoop/commit/apache-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/schema/Rule.java:[6,24]
cannot find symbol
[ERROR] symbol: class Generated
[ERROR] location: package javax.annotation
[ERROR]
/Users/stevel/hadoop/commit/apache-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/schema/Rule.java:[27,2]
cannot find symbol
[ERROR] symbol: class Generated
[ERROR]
/Users/stevel/hadoop/commit/apache-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/schema/Rule.java:[304,6]
cannot find symbol
[ERROR] symbol: class Generated
[ERROR] location: class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.schema.Rule
[ERROR]
/Users/stevel/hadoop/commit/apache-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/schema/Rule.java:[255,6]
cannot find symbol
[ERROR] symbol: class Generated
[ERROR] location: class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.schema.Rule
[ERROR]
/Users/stevel/hadoop/commit/apache-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/schema/Rule.java:[214,6]
cannot find symbol
[ERROR] symbol: class Generated
[ERROR] location: class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.schema.Rule
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-yarn-server-resourcemanager
{code}
> Create new JSON schema for Placement Rules
> --
>
> Key: YARN-10386
> URL: https://issues.apache.org/jira/browse/YARN-10386
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, capacityscheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch,
> YARN-10386-002.patch, YARN-10386-003.patch, YARN-10386-004.patch,
> YARN-10386-005.patch, YARN-10386-006.patch, YARN-10386-007.patch,
> YARN-10386-008.patch, YARN-10386-appendum.patch, YARN-10386-appendum2.patch
>
>
> Tasks in this JIRA:
> # Create new JSON schema
> # Add Maven plugin which generates Java POJOs based on the schema
> # Add helper class which essentially does the same as #2 (for dev purposes)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10967) setPermission() call floods HDFS NN RPC queue
[ https://issues.apache.org/jira/browse/YARN-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17420343#comment-17420343 ] Steve Loughran commented on YARN-10967: --- (and/or we add a getFileCountAsync(path) which returns a future with * number and size of files, no directories * IOSTats of how much that operation cost. the fact it'd be async and the cost would be exposed might make team Hive reconsider their design choices > setPermission() call floods HDFS NN RPC queue > - > > Key: YARN-10967 > URL: https://issues.apache.org/jira/browse/YARN-10967 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Anand Srinivasan >Priority: Major > Labels: performance > > Checking the code changes for the log aggregation feature, we could see that > when the log aggregator is inited for each app, we do verify and create > remote dir where we make an additional call to setPermission() even though > the remote dir exists and the permissions are set as expected. > This code path was introduced to cater to the cloud storage where we had to > make this additional check to ensure the remote file system and the > corresponding cloud storage supports setting permissions. > Upstream jira that introduced this call. > https://issues.apache.org/jira/browse/YARN-9030 > This additional setPermission() call per each app/job floods the HDFS NN and > its RPC queue which affects the performance overall. > The ask here is to see if it's feasible to do the following : > (a)if we can put the code introduced via YARN-9030 behind a configuration > option (may be setting this option to false by default (assuming the storage > used is HDFS) to bypass this code) > (b)check if customer is using HDFS storage internally in the code (by > checking yarn.nodemanager.remote-app-log-dir) and bypass this code if the > storage is indeed HDFS. > given that the code introduced in YARN-9030 is mainly put in for cloud > storage providers. > Thanks -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10967) setPermission() call floods HDFS NN RPC queue
[ https://issues.apache.org/jira/browse/YARN-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17420342#comment-17420342 ] Steve Loughran commented on YARN-10967: --- think it's probably getContentSummary which is the enemy there. I don't know what HDFS does there, but for object stores its very bad. I wish hive would stop using it, entirely. > setPermission() call floods HDFS NN RPC queue > - > > Key: YARN-10967 > URL: https://issues.apache.org/jira/browse/YARN-10967 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Anand Srinivasan >Priority: Major > Labels: performance > > Checking the code changes for the log aggregation feature, we could see that > when the log aggregator is inited for each app, we do verify and create > remote dir where we make an additional call to setPermission() even though > the remote dir exists and the permissions are set as expected. > This code path was introduced to cater to the cloud storage where we had to > make this additional check to ensure the remote file system and the > corresponding cloud storage supports setting permissions. > Upstream jira that introduced this call. > https://issues.apache.org/jira/browse/YARN-9030 > This additional setPermission() call per each app/job floods the HDFS NN and > its RPC queue which affects the performance overall. > The ask here is to see if it's feasible to do the following : > (a)if we can put the code introduced via YARN-9030 behind a configuration > option (may be setting this option to false by default (assuming the storage > used is HDFS) to bypass this code) > (b)check if customer is using HDFS storage internally in the code (by > checking yarn.nodemanager.remote-app-log-dir) and bypass this code if the > storage is indeed HDFS. > given that the code introduced in YARN-9030 is mainly put in for cloud > storage providers. > Thanks -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10967) setPermission() call floods HDFS NN RPC queue
[ https://issues.apache.org/jira/browse/YARN-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419393#comment-17419393 ] Steve Loughran commented on YARN-10967: --- 1 IO call per job overloading the NN? > setPermission() call floods HDFS NN RPC queue > - > > Key: YARN-10967 > URL: https://issues.apache.org/jira/browse/YARN-10967 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Anand Srinivasan >Priority: Major > Labels: performance > > Checking the code changes for the log aggregation feature, we could see that > when the log aggregator is inited for each app, we do verify and create > remote dir where we make an additional call to setPermission() even though > the remote dir exists and the permissions are set as expected. > This code path was introduced to cater to the cloud storage where we had to > make this additional check to ensure the remote file system and the > corresponding cloud storage supports setting permissions. > Upstream jira that introduced this call. > https://issues.apache.org/jira/browse/YARN-9030 > This additional setPermission() call per each app/job floods the HDFS NN and > its RPC queue which affects the performance overall. > The ask here is to see if it's feasible to do the following : > (a)if we can put the code introduced via YARN-9030 behind a configuration > option (may be setting this option to false by default (assuming the storage > used is HDFS) to bypass this code) > (b)check if customer is using HDFS storage internally in the code (by > checking yarn.nodemanager.remote-app-log-dir) and bypass this code if the > storage is indeed HDFS. > given that the code introduced in YARN-9030 is mainly put in for cloud > storage providers. > Thanks -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Deleted] (YARN-10964) jira test
[ https://issues.apache.org/jira/browse/YARN-10964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran deleted YARN-10964: -- > jira test > - > > Key: YARN-10964 > URL: https://issues.apache.org/jira/browse/YARN-10964 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Amit Mourya >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
