[jira] [Commented] (YARN-9278) Shuffle nodes when selecting to be preempted nodes
[
https://issues.apache.org/jira/browse/YARN-9278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796823#comment-16796823
]
Wilfred Spiegelenburg commented on YARN-9278:
-
Thank for the update [~uranus]
I don't think that we can add a lot of tests, as it would become really
difficult to inspect the results. The three tests I can think of are:
* setting the batch value to 0 (-1 is the default and we do a tests with that).
* setting the value for the batch to something larger than the number of nodes
to show that we do not run out of the node list and somehow fail
* setting the batch value to {{#NMs -1}} (i.e. batch=4 in a 5 node cluster) and
do multiple runs. Even with bad random we should flip over the end of the list.
None of the tests should fail iterating nodes.
> Shuffle nodes when selecting to be preempted nodes
> --
>
> Key: YARN-9278
> URL: https://issues.apache.org/jira/browse/YARN-9278
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Major
> Attachments: YARN-9278.001.patch, YARN-9278.002.patch,
> YARN-9278.003.patch
>
>
> We should *shuffle* the nodes to avoid some nodes being preempted frequently.
> Also, we should *limit* the num of nodes to make preemption more efficient.
> Just like this,
> {code:java}
> // we should not iterate all nodes, that will be very slow
> long maxTryNodeNum =
> context.getPreemptionConfig().getToBePreemptedNodeMaxNumOnce();
> if (potentialNodes.size() > maxTryNodeNum){
> Collections.shuffle(potentialNodes);
> List newPotentialNodes = new ArrayList();
> for (int i = 0; i < maxTryNodeNum; i++){
> newPotentialNodes.add(potentialNodes.get(i));
> }
> potentialNodes = newPotentialNodes;
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9267) General improvements in FpgaResourceHandlerImpl
[
https://issues.apache.org/jira/browse/YARN-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796800#comment-16796800
]
Devaraj K commented on YARN-9267:
-
Thanks [~pbacsko] for updating the patch.
* FpgaResourceHandlerImpl.java
** I am not sure whether this is really needed, I think {{getSha256ofFile}} can
be invoked directly and with that {{if (!hashOpt.isPresent()) {}} also can be
avoided.
{code:xml}
+ private Function> digestProvider =
+ this::getSha256ofFile;
{code}
** With the above fix, can you also update here to throw the exception directly
as a wrapped one and avoid logging.
{code:xml}
+ LOG.error("Could not calculate SHA-256", e);
{code}
* TestFpgaResourceHandler.java
** Can we have a loop here to add the {{FpgaDevice}} objects into
{{deviceList}}, so that this duplicate code can be removed.
{code:xml}
+deviceList.add(new FpgaDevice(vendorType, 247, 0, null));
.
+deviceList.add(new FpgaDevice(vendorType, 247, 4, null));
{code}
> General improvements in FpgaResourceHandlerImpl
> ---
>
> Key: YARN-9267
> URL: https://issues.apache.org/jira/browse/YARN-9267
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9267-001.patch, YARN-9267-002.patch,
> YARN-9267-003.patch, YARN-9267-004.patch, YARN-9267-005.patch,
> YARN-9267-006.patch, YARN-9267-007.patch
>
>
> Fix some problems in {{FpgaResourceHandlerImpl}}:
> * {{preStart()}} does not reconfigure card with the same IP - we see it as a
> problem. If you recompile the FPGA application, you must rename the aocx file
> because the card will not be reprogrammed. Suggestion: instead of storing
> Node<\->IPID mapping, store Node<\->IPID hash (like the SHA-256 of the
> localized file).
> * Switch to slf4j from Apache Commons Logging
> * Some unused imports
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9396) YARN_RM_CONTAINER_CREATED published twice to ATS
[
https://issues.apache.org/jira/browse/YARN-9396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796789#comment-16796789
]
Prabhu Joseph commented on YARN-9396:
-
[~eyang] Thanks for reviewing. Null check is not needed at AcquiredTransition
as it gets called only when the previous state is either NEW or ALLOCATED.
{code}
addTransition(RMContainerState.NEW, RMContainerState.ACQUIRED,
RMContainerEventType.ACQUIRED, new AcquiredTransition())
addTransition(RMContainerState.ALLOCATED, RMContainerState.ACQUIRED,
RMContainerEventType.ACQUIRED, new AcquiredTransition())
{code}
> YARN_RM_CONTAINER_CREATED published twice to ATS
>
>
> Key: YARN-9396
> URL: https://issues.apache.org/jira/browse/YARN-9396
> Project: Hadoop YARN
> Issue Type: Bug
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9396-001.patch
>
>
> RM Container Created event published twice - one from
> {{ContainerStartedTransition}} (NEW -> ALLOCATED) and another from
> {{AcquiredTransition}} (ALLOCATED -> ACQUIRED)
> {code}
> 2019-03-18 13:10:13,551 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
> container_e11_1552914589043_0001_01_01 Container Transitioned from NEW to
> ALLOCATED
> 2019-03-18 13:10:13,597 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
> container_e11_1552914589043_0001_01_01 Container Transitioned from
> ALLOCATED to ACQUIRED
> {code}
> *Duplicate Events:*
> {code}
> container_e11_1552914589043_0001_01_01 start:
> 2019-03-18 13:10:13,556 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher:
> Publishing the entity TimelineEntity[type='YARN_CONTAINER',
> id='container_e11_1552914589043_0001_01_01'], JSON-style content:
> {"metrics":[],"events":[{"id":"YARN_RM_CONTAINER_CREATED","timestamp":1552914613542,"info":{}}],"createdtime":1552914613542,"idprefix":0,"info":{"YARN_CONTAINER_ALLOCATED_PORT":45454,"YARN_CONTAINER_ALLOCATED_MEMORY":2048,"YARN_CONTAINER_ALLOCATED_PRIORITY":0,"YARN_CONTAINER_ALLOCATED_HOST":"yarn-ats-2","YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS":"http://yarn-ats-2:8042","YARN_CONTAINER_ALLOCATED_VCORE":1},"configs":{},"isrelatedto":{},"relatesto":{},"type":"YARN_CONTAINER","id":"container_e11_1552914589043_0001_01_01"}
> 2019-03-18 13:10:13,598 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher:
> Publishing the entity TimelineEntity[type='YARN_CONTAINER',
> id='container_e11_1552914589043_0001_01_01'], JSON-style content:
> {"metrics":[],"events":[{"id":"YARN_RM_CONTAINER_CREATED","timestamp":1552914613542,"info":{}}],"createdtime":1552914613542,"idprefix":0,"info":{"YARN_CONTAINER_ALLOCATED_PORT":45454,"YARN_CONTAINER_ALLOCATED_MEMORY":2048,"YARN_CONTAINER_ALLOCATED_PRIORITY":0,"YARN_CONTAINER_ALLOCATED_HOST":"yarn-ats-2","YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS":"http://yarn-ats-2:8042","YARN_CONTAINER_ALLOCATED_VCORE":1},"configs":{},"isrelatedto":{},"relatesto":{},"type":"YARN_CONTAINER","id":"container_e11_1552914589043_0001_01_01"}
> container_e11_1552914589043_0001_01_02 start:
> 2019-03-18 13:10:21,599 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher:
> Publishing the entity TimelineEntity[type='YARN_CONTAINER',
> id='container_e11_1552914589043_0001_01_02'], JSON-style content:
> {"metrics":[],"events":[{"id":"YARN_RM_CONTAINER_CREATED","timestamp":1552914621596,"info":{}}],"createdtime":1552914621596,"idprefix":0,"info":{"YARN_CONTAINER_ALLOCATED_PORT":45454,"YARN_CONTAINER_ALLOCATED_MEMORY":1024,"YARN_CONTAINER_ALLOCATED_PRIORITY":20,"YARN_CONTAINER_ALLOCATED_HOST":"yarn-ats-2","YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS":"http://yarn-ats-2:8042","YARN_CONTAINER_ALLOCATED_VCORE":1},"configs":{},"isrelatedto":{},"relatesto":{},"type":"YARN_CONTAINER","id":"container_e11_1552914589043_0001_01_02"}
> 2019-03-18 13:10:22,344 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher:
> Publishing the entity TimelineEntity[type='YARN_CONTAINER',
> id='container_e11_1552914589043_0001_01_02'], JSON-style content:
>
[jira] [Commented] (YARN-8464) Async scheduling thread could be interrupted when there are no NodeManagers in cluster
[ https://issues.apache.org/jira/browse/YARN-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796781#comment-16796781 ] Jiandan Yang commented on YARN-8464: - Hi, [~leftnoteasy], Could you give more detail analysis? I do not understand why job is accepted forever when all NMs are running? > Async scheduling thread could be interrupted when there are no NodeManagers > in cluster > -- > > Key: YARN-8464 > URL: https://issues.apache.org/jira/browse/YARN-8464 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Reporter: Charan Hebri >Assignee: Sunil Govindan >Priority: Blocker > Fix For: 3.2.0, 3.1.1 > > Attachments: YARN-8464.001.patch, YARN-8464.002.patch > > > Test scenario: > 1. Make either yarn.nodemanager.log-dirs/yarn.nodemanager.local-dirs read-only > 2. Restart NMs via Ambari, none of them show up in the RM UI as expected > 3. Revert back the read-only dirs and restart NMs > 4. Include a non-existent dir in either > yarn.nodemanager.log-dirs/yarn.nodemanager.local-dirs (1 good existing dir + > 1 non-existing dir) > 5. Restart NMs via Ambari, all NMs show as RUNNING with a Health Report > message as expected > 6. Submit a MapReduce sleep job, job goes into ACCEPTED state > 7. Job stays in ACCEPTED state forever even though all NMs are running and > have available memory > > Credits to [~charanh] who found this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9394) Use new API of RackResolver to get better performance
[ https://issues.apache.org/jira/browse/YARN-9394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796775#comment-16796775 ] Weiwei Yang commented on YARN-9394: --- Oops, looks like UT failure was caused by this patch. [~cltlfcjin], could you please take a look? > Use new API of RackResolver to get better performance > - > > Key: YARN-9394 > URL: https://issues.apache.org/jira/browse/YARN-9394 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.3.0, 3.2.1 >Reporter: Lantao Jin >Assignee: Lantao Jin >Priority: Major > Attachments: YARN-9394.001.patch > > > After adding a new API in RackResolver YARN-9332, some old callers should > switch to new API to get better performance. As an example, Spark > [YarnAllocator|https://github.com/apache/spark/blob/733f2c0b98208815f8408e36ab669d7c07e3767f/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala#L361-L363] > for Dynamic Allocation invokes > [https://github.com/apache/hadoop/blob/6fa229891e06eea62cb9634efde755f40247e816/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java#L550] > to resolve racks in a loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9394) Use new API of RackResolver to get better performance
[ https://issues.apache.org/jira/browse/YARN-9394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796770#comment-16796770 ] Weiwei Yang commented on YARN-9394: --- Thanks for the follow-up, patch LGTM. UT failure should not be related to this patch. Committing shortly. > Use new API of RackResolver to get better performance > - > > Key: YARN-9394 > URL: https://issues.apache.org/jira/browse/YARN-9394 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.3.0, 3.2.1 >Reporter: Lantao Jin >Assignee: Lantao Jin >Priority: Major > Attachments: YARN-9394.001.patch > > > After adding a new API in RackResolver YARN-9332, some old callers should > switch to new API to get better performance. As an example, Spark > [YarnAllocator|https://github.com/apache/spark/blob/733f2c0b98208815f8408e36ab669d7c07e3767f/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala#L361-L363] > for Dynamic Allocation invokes > [https://github.com/apache/hadoop/blob/6fa229891e06eea62cb9634efde755f40247e816/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java#L550] > to resolve racks in a loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9271) Backport YARN-6927 for resource type support in MapReduce
[ https://issues.apache.org/jira/browse/YARN-9271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796754#comment-16796754 ] Jonathan Hung commented on YARN-9271: - mapreduce jobclient unit tests seem to be failing in branch-2 as well: [https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/265/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt] asflicense appears to be related to MAPREDUCE-7036. > Backport YARN-6927 for resource type support in MapReduce > - > > Key: YARN-9271 > URL: https://issues.apache.org/jira/browse/YARN-9271 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9271-YARN-8200.001.patch > > > This is already in branch-3.0. Need to port it to YARN-8200/branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9271) Backport YARN-6927 for resource type support in MapReduce
[
https://issues.apache.org/jira/browse/YARN-9271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796747#comment-16796747
]
Hadoop QA commented on YARN-9271:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} YARN-8200 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
44s{color} | {color:green} YARN-8200 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
4s{color} | {color:green} YARN-8200 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
34s{color} | {color:green} YARN-8200 passed with JDK v1.8.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
56s{color} | {color:green} YARN-8200 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
47s{color} | {color:green} YARN-8200 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} YARN-8200 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
38s{color} | {color:green} YARN-8200 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} YARN-8200 passed with JDK v1.8.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
34s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
39s{color} | {color:green} the patch passed with JDK v1.8.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 7s{color} | {color:orange} root: The patch generated 7 new + 867 unchanged
- 3 fixed = 874 total (was 870) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
42s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
14s{color} | {color:green} the patch passed with JDK v1.8.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
50s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
9s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
38s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}117m 14s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
49s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}252m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |
[jira] [Commented] (YARN-9392) Handle missing scheduler events in Opportunistic Scheduler.
[ https://issues.apache.org/jira/browse/YARN-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796705#comment-16796705 ] Abhishek Modi commented on YARN-9392: - Thanks [~giovanni.fumarola] for review and committing it. > Handle missing scheduler events in Opportunistic Scheduler. > --- > > Key: YARN-9392 > URL: https://issues.apache.org/jira/browse/YARN-9392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-9392.001.patch > > > At present newly added scheduler events are not being ignored by > Opportunistic scheduler causing error messages in logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9390) Add support for configurable Resource Calculator in Opportunistic Scheduler.
[ https://issues.apache.org/jira/browse/YARN-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796704#comment-16796704 ] Abhishek Modi commented on YARN-9390: - Thanks [~giovanni.fumarola] for review. Right now, in capacity scheduler there is a configuration to change resource calculator, but there is no way to change it in Opportunistic scheduling. DefaultResourceCalculator is performant as compared to DominantResourceCalculator, as it doesn't compare other resource types. The idea is to provide flexibility and an option to the users using capacity scheduler with DefaultResourceCalculator to use opportunistic scheduling with same resource calculator. > Add support for configurable Resource Calculator in Opportunistic Scheduler. > > > Key: YARN-9390 > URL: https://issues.apache.org/jira/browse/YARN-9390 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-9390.001.patch > > > Right now, Opportunistic scheduler uses hard coded DominantResourceCalculator > and there is no option to change it to other resource calculators. This Jira > is to make resource calculator configurable for Opportunistic scheduler. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8551) Build Common module for MaWo application
[
https://issues.apache.org/jira/browse/YARN-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796644#comment-16796644
]
Hadoop QA commented on YARN-8551:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
25s{color} | {color:red} hadoop-yarn-applications-mawo in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-yarn-applications-mawo in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
30s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8551 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12963039/YARN-8551.0012.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux f65f5b89b048 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 310ebf5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/23754/artifact/out/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-mawo.txt
|
| Test Results |
[jira] [Commented] (YARN-9391) Disable PATH variable to be passed to Docker container
[ https://issues.apache.org/jira/browse/YARN-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796620#comment-16796620 ] Eric Yang commented on YARN-9391: - [~Jim_Brennan] Yes, this is the cause to add the NM variables to entrypoint docker container. > Disable PATH variable to be passed to Docker container > -- > > Key: YARN-9391 > URL: https://issues.apache.org/jira/browse/YARN-9391 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Priority: Major > > This is observed from using Apache NiFi docker image. It makes assumption > that PATH variable contains /bin to reference to system utility. Where host > YARN environment PATH variable is default to leaked into container by > accident and not containing /bin path (default configuration). In general, > it seems like node manager should block PATH variable from leaking into > container. Not sure if there is a valid use case that host PATH variable > must leak into container from docker point of view. From Hadoop point of > view, if container is merely a chroot, and container is a mirror image of > host worker dir. It is good to keep host PATH variable the same. > Maybe we want to be more specific that block PATH variable to leak into > Docker container, if it is using ENTRYPOINT only? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8551) Build Common module for MaWo application
[ https://issues.apache.org/jira/browse/YARN-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yesha Vora updated YARN-8551: - Attachment: YARN-8551.0012.patch > Build Common module for MaWo application > > > Key: YARN-8551 > URL: https://issues.apache.org/jira/browse/YARN-8551 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yesha Vora >Assignee: Yesha Vora >Priority: Major > Attachments: YARN-8551.001.patch, YARN-8551.0010.patch, > YARN-8551.0011.patch, YARN-8551.0012.patch, YARN-8551.002.patch, > YARN-8551.003.patch, YARN-8551.004.patch, YARN-8551.005.patch, > YARN-8551.006.patch, YARN-8551.007.patch, YARN-8551.008.patch, > YARN-8551.009.patch > > > Build Common module for MaWo application. > This module should include defination of Task. A Task should contain > * TaskID > * Task Command > * Task Environment > * Task Timeout > * Task Type > ** Simple Task > *** Its a single Task > ** Composite Task > *** Its a composition of multiple simple tasks > ** Teardown Task > *** Its a last task to be executed after a job is finished > ** Null Task > *** Its a null task -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9391) Disable PATH variable to be passed to Docker container
[
https://issues.apache.org/jira/browse/YARN-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796597#comment-16796597
]
Jim Brennan commented on YARN-9391:
---
[~eyang] if the concern is only for Nodemanager white-list variables leaking
through, it may be due to this code in ContainerExecutor.writeLaunchEnv():
{noformat}
// Add the whitelist vars to the environment. Do this after writing
// environment variables so they are not written twice.
for(String var : whitelistVars) {
if (!environment.containsKey(var)) {
String val = getNMEnvVar(var);
if (val != null) {
environment.put(var, val);
}
}
}
}
{noformat}
This is adding the white-listed variables to the environment map which gets
passed to launchContainer. In the native and non-entry-point cases, I don't
think this is necessary, but I am not 100% sure about that - we use the launch
script in those cases. In the entry-point case, this code is what may be adding
the white-list variables to the environment map, which you then pass raw to the
container. Note that it won't add variables that were already defined by the
user.
Do you think this might explain what you are seeing?
> Disable PATH variable to be passed to Docker container
> --
>
> Key: YARN-9391
> URL: https://issues.apache.org/jira/browse/YARN-9391
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Priority: Major
>
> This is observed from using Apache NiFi docker image. It makes assumption
> that PATH variable contains /bin to reference to system utility. Where host
> YARN environment PATH variable is default to leaked into container by
> accident and not containing /bin path (default configuration). In general,
> it seems like node manager should block PATH variable from leaking into
> container. Not sure if there is a valid use case that host PATH variable
> must leak into container from docker point of view. From Hadoop point of
> view, if container is merely a chroot, and container is a mirror image of
> host worker dir. It is good to keep host PATH variable the same.
> Maybe we want to be more specific that block PATH variable to leak into
> Docker container, if it is using ENTRYPOINT only?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8551) Build Common module for MaWo application
[
https://issues.apache.org/jira/browse/YARN-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796598#comment-16796598
]
Hadoop QA commented on YARN-8551:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo:
The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 58s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} hadoop-yarn-applications-mawo in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
18s{color} | {color:green} hadoop-yarn-applications-mawo in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
32s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8551 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12963036/YARN-8551.0011.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux d347b04a8359 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 310ebf5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| checkstyle |
[jira] [Commented] (YARN-9391) Disable PATH variable to be passed to Docker container
[ https://issues.apache.org/jira/browse/YARN-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796569#comment-16796569 ] Eric Yang commented on YARN-9391: - In non-entry point mode docker, container works more like a chroot environment. If someone wants to run mapreduce task in docker container. This would be the mode to support that use case. It is likely that host level Java and executable are available in the same location as the host. In this case, it make sense to pass in host level environment variables to docker container. If we are in agreement with the operating style of non-entrypoint docker, then we only need to make code changes for filtering for entrypoint based docker container like [~Jim_Brennan] stated. [~Jim_Brennan] would you like to take ownership of this one? > Disable PATH variable to be passed to Docker container > -- > > Key: YARN-9391 > URL: https://issues.apache.org/jira/browse/YARN-9391 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Priority: Major > > This is observed from using Apache NiFi docker image. It makes assumption > that PATH variable contains /bin to reference to system utility. Where host > YARN environment PATH variable is default to leaked into container by > accident and not containing /bin path (default configuration). In general, > it seems like node manager should block PATH variable from leaking into > container. Not sure if there is a valid use case that host PATH variable > must leak into container from docker point of view. From Hadoop point of > view, if container is merely a chroot, and container is a mirror image of > host worker dir. It is good to keep host PATH variable the same. > Maybe we want to be more specific that block PATH variable to leak into > Docker container, if it is using ENTRYPOINT only? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8551) Build Common module for MaWo application
[ https://issues.apache.org/jira/browse/YARN-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yesha Vora updated YARN-8551: - Attachment: YARN-8551.0011.patch > Build Common module for MaWo application > > > Key: YARN-8551 > URL: https://issues.apache.org/jira/browse/YARN-8551 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yesha Vora >Assignee: Yesha Vora >Priority: Major > Attachments: YARN-8551.001.patch, YARN-8551.0010.patch, > YARN-8551.0011.patch, YARN-8551.002.patch, YARN-8551.003.patch, > YARN-8551.004.patch, YARN-8551.005.patch, YARN-8551.006.patch, > YARN-8551.007.patch, YARN-8551.008.patch, YARN-8551.009.patch > > > Build Common module for MaWo application. > This module should include defination of Task. A Task should contain > * TaskID > * Task Command > * Task Environment > * Task Timeout > * Task Type > ** Simple Task > *** Its a single Task > ** Composite Task > *** Its a composition of multiple simple tasks > ** Teardown Task > *** Its a last task to be executed after a job is finished > ** Null Task > *** Its a null task -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9391) Disable PATH variable to be passed to Docker container
[ https://issues.apache.org/jira/browse/YARN-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796560#comment-16796560 ] Jim Brennan commented on YARN-9391: --- [~ebadger] I assume you are referring to the PATH variable in particular, not to all environment variables. I think in this case the PATH variable is being included in the whitelist for the NM (for the non-docker case that [~eyang] mentioned above). I agree that it is unlikely that we really need the NM's PATH variable inside of a docker container. I'm a little reluctant to have the NM just skip that whitelist entry for docker/oci runtimes. Currently the whitelist environment variable processing/script writing is done in ContainerLaunch/ContainerExecutor. What is there works for the non-entry-point case (as long as the PATH is defined in the image, which I think we are assuming here). For the entry point case, the environment is added to the run command in DockerLinuxContainerRuntime.launchContainer(). > Disable PATH variable to be passed to Docker container > -- > > Key: YARN-9391 > URL: https://issues.apache.org/jira/browse/YARN-9391 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Priority: Major > > This is observed from using Apache NiFi docker image. It makes assumption > that PATH variable contains /bin to reference to system utility. Where host > YARN environment PATH variable is default to leaked into container by > accident and not containing /bin path (default configuration). In general, > it seems like node manager should block PATH variable from leaking into > container. Not sure if there is a valid use case that host PATH variable > must leak into container from docker point of view. From Hadoop point of > view, if container is merely a chroot, and container is a mirror image of > host worker dir. It is good to keep host PATH variable the same. > Maybe we want to be more specific that block PATH variable to leak into > Docker container, if it is using ENTRYPOINT only? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9014) OCI/squashfs container runtime
[
https://issues.apache.org/jira/browse/YARN-9014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger reassigned YARN-9014:
-
Assignee: Eric Badger (was: Jason Lowe)
Assigning to myself, since Jason is no longer working on this
> OCI/squashfs container runtime
> --
>
> Key: YARN-9014
> URL: https://issues.apache.org/jira/browse/YARN-9014
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Jason Lowe
>Assignee: Eric Badger
>Priority: Major
> Labels: Docker
> Attachments: OciSquashfsRuntime.v001.pdf
>
>
> This JIRA tracks a YARN container runtime that supports running containers in
> images built by Docker but the runtime does not use Docker directly, and
> Docker does not have to be installed on the nodes. The runtime leverages the
> [OCI runtime standard|https://github.com/opencontainers/runtime-spec] to
> launch containers, so an OCI-compliant runtime like {{runc}} is required.
> {{runc}} has the benefit of not requiring a daemon like {{dockerd}} to be
> running in order to launch/control containers.
> The layers comprising the Docker image are uploaded to HDFS as
> [squashfs|http://tldp.org/HOWTO/SquashFS-HOWTO/whatis.html] images, enabling
> the runtime to efficiently download and execute directly on the compressed
> layers. This saves image unpack time and space on the local disk. The image
> layers, like other entries in the YARN distributed cache, can be spread
> across the YARN local disks, increasing the available space for storing
> container images on each node.
> A design document will be posted shortly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9391) Disable PATH variable to be passed to Docker container
[ https://issues.apache.org/jira/browse/YARN-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796533#comment-16796533 ] Eric Badger commented on YARN-9391: --- bq. I think this issue is specific to the entry-point case where whitelist variables override those specified in the image. In what circumstances does it make sense for the docker container to use the environment variables that are specified on the host? The docker image could be wildly different than the layout on the host. I don't think that we can use the assumption that the docker image and the host are going to have similar layouts. So that makes me want to not use the environment variables of the NM unless the job explicitly asks for them (or possibly not even then. Then they would just specify them in the environment for their job). > Disable PATH variable to be passed to Docker container > -- > > Key: YARN-9391 > URL: https://issues.apache.org/jira/browse/YARN-9391 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Priority: Major > > This is observed from using Apache NiFi docker image. It makes assumption > that PATH variable contains /bin to reference to system utility. Where host > YARN environment PATH variable is default to leaked into container by > accident and not containing /bin path (default configuration). In general, > it seems like node manager should block PATH variable from leaking into > container. Not sure if there is a valid use case that host PATH variable > must leak into container from docker point of view. From Hadoop point of > view, if container is merely a chroot, and container is a mirror image of > host worker dir. It is good to keep host PATH variable the same. > Maybe we want to be more specific that block PATH variable to leak into > Docker container, if it is using ENTRYPOINT only? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9391) Disable PATH variable to be passed to Docker container
[
https://issues.apache.org/jira/browse/YARN-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796521#comment-16796521
]
Jim Brennan commented on YARN-9391:
---
{quote}
The whitelist needs to behave differently for docker containers and non-docker
containers.
{quote}
[~ebadger] I'm not sure this is what we want. It already does behave
differently in that for non-Entry-Point docker, the docker image can override
whitelist variables.
I think this issue is specific to the entry-point case where whitelist
variables override those specified in the image.
> Disable PATH variable to be passed to Docker container
> --
>
> Key: YARN-9391
> URL: https://issues.apache.org/jira/browse/YARN-9391
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Priority: Major
>
> This is observed from using Apache NiFi docker image. It makes assumption
> that PATH variable contains /bin to reference to system utility. Where host
> YARN environment PATH variable is default to leaked into container by
> accident and not containing /bin path (default configuration). In general,
> it seems like node manager should block PATH variable from leaking into
> container. Not sure if there is a valid use case that host PATH variable
> must leak into container from docker point of view. From Hadoop point of
> view, if container is merely a chroot, and container is a mirror image of
> host worker dir. It is good to keep host PATH variable the same.
> Maybe we want to be more specific that block PATH variable to leak into
> Docker container, if it is using ENTRYPOINT only?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9391) Disable PATH variable to be passed to Docker container
[ https://issues.apache.org/jira/browse/YARN-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796467#comment-16796467 ] Eric Badger commented on YARN-9391: --- bq. When filtering PATH variable from environment white list, it has some undesired side effects for mapreduce style workload outside of docker container. For example, streaming task that depends on python will not work anymore. Yep, that makes sense. The whitelist needs to behave differently for docker containers and non-docker containers. > Disable PATH variable to be passed to Docker container > -- > > Key: YARN-9391 > URL: https://issues.apache.org/jira/browse/YARN-9391 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Priority: Major > > This is observed from using Apache NiFi docker image. It makes assumption > that PATH variable contains /bin to reference to system utility. Where host > YARN environment PATH variable is default to leaked into container by > accident and not containing /bin path (default configuration). In general, > it seems like node manager should block PATH variable from leaking into > container. Not sure if there is a valid use case that host PATH variable > must leak into container from docker point of view. From Hadoop point of > view, if container is merely a chroot, and container is a mirror image of > host worker dir. It is good to keep host PATH variable the same. > Maybe we want to be more specific that block PATH variable to leak into > Docker container, if it is using ENTRYPOINT only? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9271) Backport YARN-6927 for resource type support in MapReduce
[ https://issues.apache.org/jira/browse/YARN-9271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9271: Attachment: YARN-9271-YARN-8200.001.patch > Backport YARN-6927 for resource type support in MapReduce > - > > Key: YARN-9271 > URL: https://issues.apache.org/jira/browse/YARN-9271 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9271-YARN-8200.001.patch > > > This is already in branch-3.0. Need to port it to YARN-8200/branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9271) Backport YARN-6927 for resource type support in MapReduce
[ https://issues.apache.org/jira/browse/YARN-9271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796414#comment-16796414 ] Jonathan Hung commented on YARN-9271: - Attached 001 patch. It's identical to YARN-6927 but replaces some java 8 syntax when finding log lines in tests with java 7 compatible syntax: # TestTaskAttempt#testReducerMemoryRequestOverriding # TestTaskAttempt#testReducerCpuRequestOverriding # TestYARNRunner#testAMRMemoryRequestOverriding > Backport YARN-6927 for resource type support in MapReduce > - > > Key: YARN-9271 > URL: https://issues.apache.org/jira/browse/YARN-9271 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9271-YARN-8200.001.patch > > > This is already in branch-3.0. Need to port it to YARN-8200/branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9271) Backport YARN-6927 for resource type support in MapReduce
[ https://issues.apache.org/jira/browse/YARN-9271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung reassigned YARN-9271: --- Assignee: Jonathan Hung > Backport YARN-6927 for resource type support in MapReduce > - > > Key: YARN-9271 > URL: https://issues.apache.org/jira/browse/YARN-9271 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > > This is already in branch-3.0. Need to port it to YARN-8200/branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9392) Handle missing scheduler events in Opportunistic Scheduler.
[ https://issues.apache.org/jira/browse/YARN-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796381#comment-16796381 ] Hudson commented on YARN-9392: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16246 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16246/]) YARN-9392. Handle missing scheduler events in Opportunistic Scheduler. (gifuma: rev 5d8bd0e5cbf1142dbdf6a5f8cf1a448da88b9e9a) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/OpportunisticContainerAllocatorAMService.java > Handle missing scheduler events in Opportunistic Scheduler. > --- > > Key: YARN-9392 > URL: https://issues.apache.org/jira/browse/YARN-9392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-9392.001.patch > > > At present newly added scheduler events are not being ignored by > Opportunistic scheduler causing error messages in logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9392) Handle missing scheduler events in Opportunistic Scheduler.
[ https://issues.apache.org/jira/browse/YARN-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796359#comment-16796359 ] Giovanni Matteo Fumarola commented on YARN-9392: Committed to trunk. Thanks [~abmodi] > Handle missing scheduler events in Opportunistic Scheduler. > --- > > Key: YARN-9392 > URL: https://issues.apache.org/jira/browse/YARN-9392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-9392.001.patch > > > At present newly added scheduler events are not being ignored by > Opportunistic scheduler causing error messages in logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9390) Add support for configurable Resource Calculator in Opportunistic Scheduler.
[ https://issues.apache.org/jira/browse/YARN-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796361#comment-16796361 ] Giovanni Matteo Fumarola commented on YARN-9390: Thanks [~abmodi]. Why do we need this? > Add support for configurable Resource Calculator in Opportunistic Scheduler. > > > Key: YARN-9390 > URL: https://issues.apache.org/jira/browse/YARN-9390 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-9390.001.patch > > > Right now, Opportunistic scheduler uses hard coded DominantResourceCalculator > and there is no option to change it to other resource calculators. This Jira > is to make resource calculator configurable for Opportunistic scheduler. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9358) Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322
[ https://issues.apache.org/jira/browse/YARN-9358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796305#comment-16796305 ] Daniel Templeton commented on YARN-9358: I just did a super quick pass. There are a couple issues in basic formatting. The descriptions must all end in a period. Otherwise it's a javadoc error. Also, the param and returns tag descriptions should not start with a capital letter nor end with a period (I don't remember seeing extraneous periods, but just in case...) > Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322 > -- > > Key: YARN-9358 > URL: https://issues.apache.org/jira/browse/YARN-9358 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-9358.001.patch, YARN-9358.002.patch, > YARN-9358.003.patch, YARN-9358.004.patch > > > This is a follow-up for YARN-9322, covering javadoc changes as discussed with > [~templedf] earlier. > As discussed with Daniel, we need to add javadoc for the new methods > introduced with YARN-9322 and also for the modified methods. > The javadoc should refer to the fact that Resource Types are also included in > the Resource object in case of get/set as well. > The methods are: > 1. getFairShare / setFairShare > 2. getSteadyFairShare / setSteadyFairShare > 3. getMinShare / setMinShare > 4. getMaxShare / setMaxShare > 5. getMaxAMShare / setMaxAMShare > 6. getAMResourceUsage / setAMResourceUsage > Moreover, a javadoc could be added to the constructor of FSQueueMetrics as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9358) Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322
[
https://issues.apache.org/jira/browse/YARN-9358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796295#comment-16796295
]
Hadoop QA commented on YARN-9358:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 21 unchanged - 0 fixed = 22 total (was 21) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 76m
15s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}123m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9358 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12962995/YARN-9358.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 398a9be85a25 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c9e50c4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/23750/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23750/testReport/ |
| Max. process+thread count | 892
[jira] [Commented] (YARN-9399) Yarn Client may use stale DNS to connect to RM
[ https://issues.apache.org/jira/browse/YARN-9399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796289#comment-16796289 ] Íñigo Goiri commented on YARN-9399: --- This is somewhat related to HDFS-4957 in the HDFS side. The discussion seems pretty related jere: https://github.com/apache-spark-on-k8s/kubernetes-HDFS/issues/48 [~fengnanli], in HDFS-14327 you are using FQDN addresses. Should we cover this scenario there? > Yarn Client may use stale DNS to connect to RM > -- > > Key: YARN-9399 > URL: https://issues.apache.org/jira/browse/YARN-9399 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.1 >Reporter: Leon zhang >Assignee: Íñigo Goiri >Priority: Major > Labels: patch > Original Estimate: 168h > Remaining Estimate: 168h > > This happens more frequently when running yarn in Kubernetes. When yarn > client try to connect to RM, if the DNS of RM is not resovable due to > kube-dns failure or not ready, the yarn client will initaize itself with > unresoved InetSocketAddress in RMProxy#newProxyInstance(). The connect to RM > will fail with UnknownHostException. Yarn client will retry the connection by > RetryProxy by it always use the cached unresolved InetSocketAddress. The > retry will never success. When RM is reschdured to another kubernetes node, > which changed the RM ip, this bug will also happen. Currently the work around > is to restarting the Yarn client. > This issue happens in both HA and non-HA of RM. HDFS has simialr issues. > [https://github.com/apache-spark-on-k8s/kubernetes-HDFS/issues/48] > I propose to add a new RMFailoverProxyProvider called > AutoRefreshRMFailoverProxyProvider which will resove the DNS in the > overwriten function getProxy(). This way, RetryProxy can resolve the DNS each > time it retry. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9396) YARN_RM_CONTAINER_CREATED published twice to ATS
[
https://issues.apache.org/jira/browse/YARN-9396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796252#comment-16796252
]
Eric Yang commented on YARN-9396:
-
[~Prabhu Joseph] Do we need to check if container.getState() != null as part of
the condition check? Some code in RM indicate that container.getState() can be
null in some situations.
> YARN_RM_CONTAINER_CREATED published twice to ATS
>
>
> Key: YARN-9396
> URL: https://issues.apache.org/jira/browse/YARN-9396
> Project: Hadoop YARN
> Issue Type: Bug
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9396-001.patch
>
>
> RM Container Created event published twice - one from
> {{ContainerStartedTransition}} (NEW -> ALLOCATED) and another from
> {{AcquiredTransition}} (ALLOCATED -> ACQUIRED)
> {code}
> 2019-03-18 13:10:13,551 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
> container_e11_1552914589043_0001_01_01 Container Transitioned from NEW to
> ALLOCATED
> 2019-03-18 13:10:13,597 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
> container_e11_1552914589043_0001_01_01 Container Transitioned from
> ALLOCATED to ACQUIRED
> {code}
> *Duplicate Events:*
> {code}
> container_e11_1552914589043_0001_01_01 start:
> 2019-03-18 13:10:13,556 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher:
> Publishing the entity TimelineEntity[type='YARN_CONTAINER',
> id='container_e11_1552914589043_0001_01_01'], JSON-style content:
> {"metrics":[],"events":[{"id":"YARN_RM_CONTAINER_CREATED","timestamp":1552914613542,"info":{}}],"createdtime":1552914613542,"idprefix":0,"info":{"YARN_CONTAINER_ALLOCATED_PORT":45454,"YARN_CONTAINER_ALLOCATED_MEMORY":2048,"YARN_CONTAINER_ALLOCATED_PRIORITY":0,"YARN_CONTAINER_ALLOCATED_HOST":"yarn-ats-2","YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS":"http://yarn-ats-2:8042","YARN_CONTAINER_ALLOCATED_VCORE":1},"configs":{},"isrelatedto":{},"relatesto":{},"type":"YARN_CONTAINER","id":"container_e11_1552914589043_0001_01_01"}
> 2019-03-18 13:10:13,598 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher:
> Publishing the entity TimelineEntity[type='YARN_CONTAINER',
> id='container_e11_1552914589043_0001_01_01'], JSON-style content:
> {"metrics":[],"events":[{"id":"YARN_RM_CONTAINER_CREATED","timestamp":1552914613542,"info":{}}],"createdtime":1552914613542,"idprefix":0,"info":{"YARN_CONTAINER_ALLOCATED_PORT":45454,"YARN_CONTAINER_ALLOCATED_MEMORY":2048,"YARN_CONTAINER_ALLOCATED_PRIORITY":0,"YARN_CONTAINER_ALLOCATED_HOST":"yarn-ats-2","YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS":"http://yarn-ats-2:8042","YARN_CONTAINER_ALLOCATED_VCORE":1},"configs":{},"isrelatedto":{},"relatesto":{},"type":"YARN_CONTAINER","id":"container_e11_1552914589043_0001_01_01"}
> container_e11_1552914589043_0001_01_02 start:
> 2019-03-18 13:10:21,599 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher:
> Publishing the entity TimelineEntity[type='YARN_CONTAINER',
> id='container_e11_1552914589043_0001_01_02'], JSON-style content:
> {"metrics":[],"events":[{"id":"YARN_RM_CONTAINER_CREATED","timestamp":1552914621596,"info":{}}],"createdtime":1552914621596,"idprefix":0,"info":{"YARN_CONTAINER_ALLOCATED_PORT":45454,"YARN_CONTAINER_ALLOCATED_MEMORY":1024,"YARN_CONTAINER_ALLOCATED_PRIORITY":20,"YARN_CONTAINER_ALLOCATED_HOST":"yarn-ats-2","YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS":"http://yarn-ats-2:8042","YARN_CONTAINER_ALLOCATED_VCORE":1},"configs":{},"isrelatedto":{},"relatesto":{},"type":"YARN_CONTAINER","id":"container_e11_1552914589043_0001_01_02"}
> 2019-03-18 13:10:22,344 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher:
> Publishing the entity TimelineEntity[type='YARN_CONTAINER',
> id='container_e11_1552914589043_0001_01_02'], JSON-style content:
> {"metrics":[],"events":[{"id":"YARN_RM_CONTAINER_CREATED","timestamp":1552914621596,"info":{}}],"createdtime":1552914621596,"idprefix":0,"info":{"YARN_CONTAINER_ALLOCATED_PORT":45454,"YARN_CONTAINER_ALLOCATED_MEMORY":1024,"YARN_CONTAINER_ALLOCATED_PRIORITY":20,"YARN_CONTAINER_ALLOCATED_HOST":"yarn-ats-2","YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS":"http://yarn-ats-2:8042","YARN_CONTAINER_ALLOCATED_VCORE":1},"configs":{},"isrelatedto":{},"relatesto":{},"type":"YARN_CONTAINER","id":"container_e11_1552914589043_0001_01_02"}
> container_e11_1552914589043_0001_01_03 start:
> 2019-03-18 13:10:27,918 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher:
> Publishing the entity
[jira] [Comment Edited] (YARN-9398) Javadoc error on FPGA related java files
[
https://issues.apache.org/jira/browse/YARN-9398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796244#comment-16796244
]
Eric Yang edited comment on YARN-9398 at 3/19/19 4:25 PM:
--
[~templedf] There is only one that is introduced 6 days ago that breaks the
build, which is the last item on my list. The other are warning messages but
the build can still succeed.
{code}
[ERROR]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/IntelFpgaOpenclPlugin.java:156:
error: bad HTML entity
[ERROR]* Helper class to run aocl diagnose & determine major/minor numbers.
{code}
I didn't capture all of the warnings, but a subset of them that were introduced
in YARN-9266.
was (Author: eyang):
[~templedf] There is only one that is introduced 6 days ago that breaks the
build, which is the last item on my list. The other are warning messages but
the build can still succeed.
{code}
[ERROR]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/IntelFpgaOpenclPlugin.java:156:
error: bad HTML entity
[ERROR]* Helper class to run aocl diagnose & determine major/minor numbers.
{code}
I didn't capture all of the errors, but a subset of them that were introduced
in YARN-9266.
> Javadoc error on FPGA related java files
>
>
> Key: YARN-9398
> URL: https://issues.apache.org/jira/browse/YARN-9398
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Peter Bacsko
>Priority: Major
>
> {code}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @param for conf
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @return
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @param for timeout
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @return
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:64:
> warning: no @return
> [ERROR] String getFpgaType();
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @return
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @throws for
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerException
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/IntelFpgaOpenclPlugin.java:156:
> error: bad HTML entity
> [ERROR]* Helper class to run aocl diagnose & determine major/minor
> numbers.
> {code}
> YARN-9266 introduced some javadoc compilation errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (YARN-9398) Javadoc error on FPGA related java files
[
https://issues.apache.org/jira/browse/YARN-9398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796244#comment-16796244
]
Eric Yang commented on YARN-9398:
-
[~templedf] There is only one that is introduced 6 days ago that breaks the
build, which is the last item on my list. The other are warning messages but
the build can still succeed.
{code}
[ERROR]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/IntelFpgaOpenclPlugin.java:156:
error: bad HTML entity
[ERROR]* Helper class to run aocl diagnose & determine major/minor numbers.
{code}
I didn't capture all of the errors, but a subset of them that were introduced
in YARN-9266.
> Javadoc error on FPGA related java files
>
>
> Key: YARN-9398
> URL: https://issues.apache.org/jira/browse/YARN-9398
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Peter Bacsko
>Priority: Major
>
> {code}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @param for conf
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @return
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @param for timeout
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @return
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:64:
> warning: no @return
> [ERROR] String getFpgaType();
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @return
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @throws for
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerException
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/IntelFpgaOpenclPlugin.java:156:
> error: bad HTML entity
> [ERROR]* Helper class to run aocl diagnose & determine major/minor
> numbers.
> {code}
> YARN-9266 introduced some javadoc compilation errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9391) Disable PATH variable to be passed to Docker container
[ https://issues.apache.org/jira/browse/YARN-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796238#comment-16796238 ] Eric Yang commented on YARN-9391: - [~ebadger] When filtering PATH variable from environment white list, it has some undesired side effects for mapreduce style workload outside of docker container. For example, streaming task that depends on python will not work anymore. If we are looking at the problem by partition of container types, the desired outcome looks more like this: | | Linux Container | Docker without EntryPoint | Docker with EntryPoint | | Allowed variables | All white listed variables | All white listed variables + Docker specific variables + YARN User defined variables | Subset of white listed variables + YARN Docker specific variables + User defined variables | | Shell expansion of variables | Yes | Yes | No | It looks like the subset of variables to pass for entrypoint mode is LANG and TZ only . The rest will have undesired side effects. > Disable PATH variable to be passed to Docker container > -- > > Key: YARN-9391 > URL: https://issues.apache.org/jira/browse/YARN-9391 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Priority: Major > > This is observed from using Apache NiFi docker image. It makes assumption > that PATH variable contains /bin to reference to system utility. Where host > YARN environment PATH variable is default to leaked into container by > accident and not containing /bin path (default configuration). In general, > it seems like node manager should block PATH variable from leaking into > container. Not sure if there is a valid use case that host PATH variable > must leak into container from docker point of view. From Hadoop point of > view, if container is merely a chroot, and container is a mirror image of > host worker dir. It is good to keep host PATH variable the same. > Maybe we want to be more specific that block PATH variable to leak into > Docker container, if it is using ENTRYPOINT only? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9335) [atsv2] Restrict the number of elements held in NM timeline collector when backend is unreachable for async calls
[
https://issues.apache.org/jira/browse/YARN-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796232#comment-16796232
]
Hadoop QA commented on YARN-9335:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 33s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
50s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
46s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
24s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9335 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12962988/YARN-9335.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 968f12e2e534 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-9391) Disable PATH variable to be passed to Docker container
[
https://issues.apache.org/jira/browse/YARN-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796227#comment-16796227
]
Jim Brennan commented on YARN-9391:
---
[~ebadger] you are correct. It doesn't look like we explicitly add PATH to the
container environment unless is is specified in {{yarn.nodemanager.admin-env}}
or, as you say, if it is specified in {{yarn.nodemanager.env-whitelist}}.
[~eyang] do you know where the PATH variable is coming from in this case?
> Disable PATH variable to be passed to Docker container
> --
>
> Key: YARN-9391
> URL: https://issues.apache.org/jira/browse/YARN-9391
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Priority: Major
>
> This is observed from using Apache NiFi docker image. It makes assumption
> that PATH variable contains /bin to reference to system utility. Where host
> YARN environment PATH variable is default to leaked into container by
> accident and not containing /bin path (default configuration). In general,
> it seems like node manager should block PATH variable from leaking into
> container. Not sure if there is a valid use case that host PATH variable
> must leak into container from docker point of view. From Hadoop point of
> view, if container is merely a chroot, and container is a mirror image of
> host worker dir. It is good to keep host PATH variable the same.
> Maybe we want to be more specific that block PATH variable to leak into
> Docker container, if it is using ENTRYPOINT only?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9391) Disable PATH variable to be passed to Docker container
[ https://issues.apache.org/jira/browse/YARN-9391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796216#comment-16796216 ] Eric Badger commented on YARN-9391: --- We can already do this today by changing the whitelist to not include PATH, right [~Jim_Brennan]? I agree with [~eyang] that the PATH variable (or anything really) outside of the container shouldn't really be relevant inside of the container. Ideally, the image should define PATH so that it will override what the NM has. But, in the case that it isn't set, I'm not sure falling back to the NM PATH is the correct thing to do. At the best it's masking failures and at the worst it's leaking environment variable info about the host. And just a note, if PATH is set in the image, it will be selected over what is set in the whitelist. The only way this isn't true is if the variable was explicitly set by the user. [~Jim_Brennan] can correct me if I'm wrong on this. > Disable PATH variable to be passed to Docker container > -- > > Key: YARN-9391 > URL: https://issues.apache.org/jira/browse/YARN-9391 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Priority: Major > > This is observed from using Apache NiFi docker image. It makes assumption > that PATH variable contains /bin to reference to system utility. Where host > YARN environment PATH variable is default to leaked into container by > accident and not containing /bin path (default configuration). In general, > it seems like node manager should block PATH variable from leaking into > container. Not sure if there is a valid use case that host PATH variable > must leak into container from docker point of view. From Hadoop point of > view, if container is merely a chroot, and container is a mirror image of > host worker dir. It is good to keep host PATH variable the same. > Maybe we want to be more specific that block PATH variable to leak into > Docker container, if it is using ENTRYPOINT only? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9358) Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322
[ https://issues.apache.org/jira/browse/YARN-9358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796185#comment-16796185 ] Zoltan Siegl commented on YARN-9358: [~templedf] could you please check this? > Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322 > -- > > Key: YARN-9358 > URL: https://issues.apache.org/jira/browse/YARN-9358 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-9358.001.patch, YARN-9358.002.patch, > YARN-9358.003.patch, YARN-9358.004.patch > > > This is a follow-up for YARN-9322, covering javadoc changes as discussed with > [~templedf] earlier. > As discussed with Daniel, we need to add javadoc for the new methods > introduced with YARN-9322 and also for the modified methods. > The javadoc should refer to the fact that Resource Types are also included in > the Resource object in case of get/set as well. > The methods are: > 1. getFairShare / setFairShare > 2. getSteadyFairShare / setSteadyFairShare > 3. getMinShare / setMinShare > 4. getMaxShare / setMaxShare > 5. getMaxAMShare / setMaxAMShare > 6. getAMResourceUsage / setAMResourceUsage > Moreover, a javadoc could be added to the constructor of FSQueueMetrics as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9358) Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322
[ https://issues.apache.org/jira/browse/YARN-9358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796178#comment-16796178 ] Zoltan Siegl commented on YARN-9358: [~pbacsko] thank you for the review, I have changed those accordingly to "fair share of the queue" and "The passed resource object". These sound even better. :) > Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322 > -- > > Key: YARN-9358 > URL: https://issues.apache.org/jira/browse/YARN-9358 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-9358.001.patch, YARN-9358.002.patch, > YARN-9358.003.patch, YARN-9358.004.patch > > > This is a follow-up for YARN-9322, covering javadoc changes as discussed with > [~templedf] earlier. > As discussed with Daniel, we need to add javadoc for the new methods > introduced with YARN-9322 and also for the modified methods. > The javadoc should refer to the fact that Resource Types are also included in > the Resource object in case of get/set as well. > The methods are: > 1. getFairShare / setFairShare > 2. getSteadyFairShare / setSteadyFairShare > 3. getMinShare / setMinShare > 4. getMaxShare / setMaxShare > 5. getMaxAMShare / setMaxAMShare > 6. getAMResourceUsage / setAMResourceUsage > Moreover, a javadoc could be added to the constructor of FSQueueMetrics as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9358) Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322
[ https://issues.apache.org/jira/browse/YARN-9358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl updated YARN-9358: --- Attachment: YARN-9358.004.patch > Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322 > -- > > Key: YARN-9358 > URL: https://issues.apache.org/jira/browse/YARN-9358 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-9358.001.patch, YARN-9358.002.patch, > YARN-9358.003.patch, YARN-9358.004.patch > > > This is a follow-up for YARN-9322, covering javadoc changes as discussed with > [~templedf] earlier. > As discussed with Daniel, we need to add javadoc for the new methods > introduced with YARN-9322 and also for the modified methods. > The javadoc should refer to the fact that Resource Types are also included in > the Resource object in case of get/set as well. > The methods are: > 1. getFairShare / setFairShare > 2. getSteadyFairShare / setSteadyFairShare > 3. getMinShare / setMinShare > 4. getMaxShare / setMaxShare > 5. getMaxAMShare / setMaxAMShare > 6. getAMResourceUsage / setAMResourceUsage > Moreover, a javadoc could be added to the constructor of FSQueueMetrics as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9358) Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322
[
https://issues.apache.org/jira/browse/YARN-9358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796166#comment-16796166
]
Hadoop QA commented on YARN-9358:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 21 unchanged - 0 fixed = 22 total (was 21) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 41s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9358 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12962968/YARN-9358.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9d6502478c7a 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c9e50c4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/23748/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
[jira] [Commented] (YARN-9358) Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322
[
https://issues.apache.org/jira/browse/YARN-9358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796143#comment-16796143
]
Peter Bacsko commented on YARN-9358:
Just minor things:
* "fair share for queue" --> I think "fair share of a queue" sounds better
* "The {@link Resource} object given" --> "The given resource object" sounds
better
> Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322
> --
>
> Key: YARN-9358
> URL: https://issues.apache.org/jira/browse/YARN-9358
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Zoltan Siegl
>Priority: Major
> Attachments: YARN-9358.001.patch, YARN-9358.002.patch,
> YARN-9358.003.patch
>
>
> This is a follow-up for YARN-9322, covering javadoc changes as discussed with
> [~templedf] earlier.
> As discussed with Daniel, we need to add javadoc for the new methods
> introduced with YARN-9322 and also for the modified methods.
> The javadoc should refer to the fact that Resource Types are also included in
> the Resource object in case of get/set as well.
> The methods are:
> 1. getFairShare / setFairShare
> 2. getSteadyFairShare / setSteadyFairShare
> 3. getMinShare / setMinShare
> 4. getMaxShare / setMaxShare
> 5. getMaxAMShare / setMaxAMShare
> 6. getAMResourceUsage / setAMResourceUsage
> Moreover, a javadoc could be added to the constructor of FSQueueMetrics as
> well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9335) [atsv2] Restrict the number of elements held in NM timeline collector when backend is unreachable for async calls
[ https://issues.apache.org/jira/browse/YARN-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-9335: Attachment: YARN-9335.003.patch > [atsv2] Restrict the number of elements held in NM timeline collector when > backend is unreachable for async calls > - > > Key: YARN-9335 > URL: https://issues.apache.org/jira/browse/YARN-9335 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vrushali C >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-9335.001.patch, YARN-9335.002.patch, > YARN-9335.003.patch > > > For ATSv2 , if the backend is unreachable, the number/size of data held in > timeline collector's memory increases significantly. This is not good for the > NM memory. > Filing jira to set a limit on how many/much should be retained by the > timeline collector in memory in case the backend is not reachable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9335) [atsv2] Restrict the number of elements held in NM timeline collector when backend is unreachable for async calls
[ https://issues.apache.org/jira/browse/YARN-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796142#comment-16796142 ] Abhishek Modi commented on YARN-9335: - Thanks [~sadineni] for pointing it out. Attached 003 patch with the fix. > [atsv2] Restrict the number of elements held in NM timeline collector when > backend is unreachable for async calls > - > > Key: YARN-9335 > URL: https://issues.apache.org/jira/browse/YARN-9335 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vrushali C >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-9335.001.patch, YARN-9335.002.patch, > YARN-9335.003.patch > > > For ATSv2 , if the backend is unreachable, the number/size of data held in > timeline collector's memory increases significantly. This is not good for the > NM memory. > Filing jira to set a limit on how many/much should be retained by the > timeline collector in memory in case the backend is not reachable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9356) Add more tests to ratio method in TestResourceCalculator
[
https://issues.apache.org/jira/browse/YARN-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796072#comment-16796072
]
Hadoop QA commented on YARN-9356:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 40s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
36s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9356 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12962965/YARN-9356.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d6db0bf734ba 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c9e50c4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23747/testReport/ |
| Max. process+thread count | 434 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/23747/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add more tests to ratio method in TestResourceCalculator
> -
>
>
[jira] [Updated] (YARN-9358) Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322
[ https://issues.apache.org/jira/browse/YARN-9358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl updated YARN-9358: --- Attachment: (was: YARN-9358_003.patch) > Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322 > -- > > Key: YARN-9358 > URL: https://issues.apache.org/jira/browse/YARN-9358 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-9358.001.patch, YARN-9358.002.patch, > YARN-9358.003.patch > > > This is a follow-up for YARN-9322, covering javadoc changes as discussed with > [~templedf] earlier. > As discussed with Daniel, we need to add javadoc for the new methods > introduced with YARN-9322 and also for the modified methods. > The javadoc should refer to the fact that Resource Types are also included in > the Resource object in case of get/set as well. > The methods are: > 1. getFairShare / setFairShare > 2. getSteadyFairShare / setSteadyFairShare > 3. getMinShare / setMinShare > 4. getMaxShare / setMaxShare > 5. getMaxAMShare / setMaxAMShare > 6. getAMResourceUsage / setAMResourceUsage > Moreover, a javadoc could be added to the constructor of FSQueueMetrics as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9358) Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322
[ https://issues.apache.org/jira/browse/YARN-9358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796024#comment-16796024 ] Zoltan Siegl commented on YARN-9358: [~adam.antal] Thanks for the review. This time I really fixed all of your review items. ;) > Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322 > -- > > Key: YARN-9358 > URL: https://issues.apache.org/jira/browse/YARN-9358 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-9358.001.patch, YARN-9358.002.patch, > YARN-9358.003.patch > > > This is a follow-up for YARN-9322, covering javadoc changes as discussed with > [~templedf] earlier. > As discussed with Daniel, we need to add javadoc for the new methods > introduced with YARN-9322 and also for the modified methods. > The javadoc should refer to the fact that Resource Types are also included in > the Resource object in case of get/set as well. > The methods are: > 1. getFairShare / setFairShare > 2. getSteadyFairShare / setSteadyFairShare > 3. getMinShare / setMinShare > 4. getMaxShare / setMaxShare > 5. getMaxAMShare / setMaxAMShare > 6. getAMResourceUsage / setAMResourceUsage > Moreover, a javadoc could be added to the constructor of FSQueueMetrics as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9358) Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322
[ https://issues.apache.org/jira/browse/YARN-9358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl updated YARN-9358: --- Attachment: YARN-9358_003.patch > Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322 > -- > > Key: YARN-9358 > URL: https://issues.apache.org/jira/browse/YARN-9358 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-9358.001.patch, YARN-9358.002.patch, > YARN-9358.003.patch > > > This is a follow-up for YARN-9322, covering javadoc changes as discussed with > [~templedf] earlier. > As discussed with Daniel, we need to add javadoc for the new methods > introduced with YARN-9322 and also for the modified methods. > The javadoc should refer to the fact that Resource Types are also included in > the Resource object in case of get/set as well. > The methods are: > 1. getFairShare / setFairShare > 2. getSteadyFairShare / setSteadyFairShare > 3. getMinShare / setMinShare > 4. getMaxShare / setMaxShare > 5. getMaxAMShare / setMaxAMShare > 6. getAMResourceUsage / setAMResourceUsage > Moreover, a javadoc could be added to the constructor of FSQueueMetrics as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9358) Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322
[ https://issues.apache.org/jira/browse/YARN-9358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl updated YARN-9358: --- Attachment: YARN-9358.003.patch > Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322 > -- > > Key: YARN-9358 > URL: https://issues.apache.org/jira/browse/YARN-9358 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-9358.001.patch, YARN-9358.002.patch, > YARN-9358.003.patch > > > This is a follow-up for YARN-9322, covering javadoc changes as discussed with > [~templedf] earlier. > As discussed with Daniel, we need to add javadoc for the new methods > introduced with YARN-9322 and also for the modified methods. > The javadoc should refer to the fact that Resource Types are also included in > the Resource object in case of get/set as well. > The methods are: > 1. getFairShare / setFairShare > 2. getSteadyFairShare / setSteadyFairShare > 3. getMinShare / setMinShare > 4. getMaxShare / setMaxShare > 5. getMaxAMShare / setMaxAMShare > 6. getAMResourceUsage / setAMResourceUsage > Moreover, a javadoc could be added to the constructor of FSQueueMetrics as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9356) Add more tests to ratio method in TestResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl updated YARN-9356: --- Attachment: YARN-9356.001.patch > Add more tests to ratio method in TestResourceCalculator > - > > Key: YARN-9356 > URL: https://issues.apache.org/jira/browse/YARN-9356 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Priority: Major > Attachments: YARN-9356.001.patch > > > TestResourceCalculator has some edge-case testcases to verify how division by > zero is handled with ResourceCalculator. > We need other basic tests like we have for other ResourceCalculator methods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9356) Add more tests to ratio method in TestResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl reassigned YARN-9356: -- Assignee: Zoltan Siegl > Add more tests to ratio method in TestResourceCalculator > - > > Key: YARN-9356 > URL: https://issues.apache.org/jira/browse/YARN-9356 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-9356.001.patch > > > TestResourceCalculator has some edge-case testcases to verify how division by > zero is handled with ResourceCalculator. > We need other basic tests like we have for other ResourceCalculator methods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9267) General improvements in FpgaResourceHandlerImpl
[
https://issues.apache.org/jira/browse/YARN-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796011#comment-16796011
]
Peter Bacsko commented on YARN-9267:
[~devaraj.k] I think now you can submit the patch.
> General improvements in FpgaResourceHandlerImpl
> ---
>
> Key: YARN-9267
> URL: https://issues.apache.org/jira/browse/YARN-9267
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9267-001.patch, YARN-9267-002.patch,
> YARN-9267-003.patch, YARN-9267-004.patch, YARN-9267-005.patch,
> YARN-9267-006.patch, YARN-9267-007.patch
>
>
> Fix some problems in {{FpgaResourceHandlerImpl}}:
> * {{preStart()}} does not reconfigure card with the same IP - we see it as a
> problem. If you recompile the FPGA application, you must rename the aocx file
> because the card will not be reprogrammed. Suggestion: instead of storing
> Node<\->IPID mapping, store Node<\->IPID hash (like the SHA-256 of the
> localized file).
> * Switch to slf4j from Apache Commons Logging
> * Some unused imports
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9398) Javadoc error on FPGA related java files
[
https://issues.apache.org/jira/browse/YARN-9398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795995#comment-16795995
]
Daniel Templeton commented on YARN-9398:
I'm seeing a ton of errors in the javadoc when I do a dist build, and they're
not the ones you're seeing:
{noformat}[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:jar (module-javadocs) on
project hadoop-yarn-server-nodemanager: MavenReportException: Error while
generating Javadoc:
[ERROR] Exit code: 1 - Picked up _JAVA_OPTIONS: -Djava.awt.headless=true
[ERROR]
/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java:392:
warning: no @param for credentials
[ERROR] protected void initializePipeline(ApplicationAttemptId
applicationAttemptId,
[ERROR] ^
[ERROR]
/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/FederationInterceptor.java:962:
warning: no @param for homeResponse
[ERROR] protected void reAttachUAMAndMergeRegisterResponse(
[ERROR] ^
[ERROR]
/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/FederationInterceptor.java:962:
warning: no @param for appId
[ERROR] protected void reAttachUAMAndMergeRegisterResponse(
[ERROR] ^
[ERROR]
/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/FederationInterceptor.java:1302:
warning: no @return
[ERROR] protected AllocateResponse generateBaseAllocationResponse() {
[ERROR] ^
[ERROR]
/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DevicePlugin.java:30:
warning: no description for @throws
[ERROR]* @throws Exception
[ERROR] ^
...
[ERROR]
[ERROR] Command line was:
/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre/../bin/javadoc
@options @packages
[ERROR]
[ERROR] Refer to the generated Javadoc files in
'/Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/apidocs'
dir.
[ERROR]
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-yarn-server-nodemanager
{noformat}
I never build with javadocs, so maybe those issues have been there a while
without me noticing. How come I don't see the errors shown in this JIRA but
instead see different errors?
> Javadoc error on FPGA related java files
>
>
> Key: YARN-9398
> URL: https://issues.apache.org/jira/browse/YARN-9398
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Peter Bacsko
>Priority: Major
>
> {code}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @param for conf
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @return
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @param for timeout
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
>
[jira] [Commented] (YARN-9267) General improvements in FpgaResourceHandlerImpl
[
https://issues.apache.org/jira/browse/YARN-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795981#comment-16795981
]
Hadoop QA commented on YARN-9267:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 0 new + 46 unchanged - 6 fixed = 46 total (was 52) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
2s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9267 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12962954/YARN-9267-007.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 850250afbd93 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c9e50c4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23745/testReport/ |
| Max. process+thread count | 413 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/23745/console |
| Powered by | Apache Yetus
[jira] [Commented] (YARN-9268) General improvements in FpgaDevice
[
https://issues.apache.org/jira/browse/YARN-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795941#comment-16795941
]
Hadoop QA commented on YARN-9268:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-9268 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9268 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12962959/YARN-9268-004.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/23746/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> General improvements in FpgaDevice
> --
>
> Key: YARN-9268
> URL: https://issues.apache.org/jira/browse/YARN-9268
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9268-001.patch, YARN-9268-002.patch,
> YARN-9268-003.patch, YARN-9268-004.patch
>
>
> Need to fix the following in the class {{FpgaDevice}}:
> * It implements {{Comparable}}, but returns 0 in every case. There is no
> natural ordering among FPGA devices, perhaps "acl0" comes before "acl1", but
> this seems too forced and unnecessary.We think this class should not
> implement {{Comparable}} at all, at least not like that.
> * Stores unnecessary fields: devName, busNum, temperature, power usage. For
> one, these are never needed in the code. Secondly, temp and power usage
> changes constantly. It's pointless to store these in this POJO.
> * {{serialVersionUID}} is 1L - let's generate a number for this
> * Use {{int}} instead of {{Integer}} - don't allow nulls. If major/minor
> uniquely identifies the card, then let's demand them in the constructor and
> don't store Integers that can be null.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9268) General improvements in FpgaDevice
[
https://issues.apache.org/jira/browse/YARN-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795944#comment-16795944
]
Peter Bacsko commented on YARN-9268:
Just as I thought, it fails to apply. It should be good after YARN-9267.
> General improvements in FpgaDevice
> --
>
> Key: YARN-9268
> URL: https://issues.apache.org/jira/browse/YARN-9268
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9268-001.patch, YARN-9268-002.patch,
> YARN-9268-003.patch, YARN-9268-004.patch
>
>
> Need to fix the following in the class {{FpgaDevice}}:
> * It implements {{Comparable}}, but returns 0 in every case. There is no
> natural ordering among FPGA devices, perhaps "acl0" comes before "acl1", but
> this seems too forced and unnecessary.We think this class should not
> implement {{Comparable}} at all, at least not like that.
> * Stores unnecessary fields: devName, busNum, temperature, power usage. For
> one, these are never needed in the code. Secondly, temp and power usage
> changes constantly. It's pointless to store these in this POJO.
> * {{serialVersionUID}} is 1L - let's generate a number for this
> * Use {{int}} instead of {{Integer}} - don't allow nulls. If major/minor
> uniquely identifies the card, then let's demand them in the constructor and
> don't store Integers that can be null.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9268) General improvements in FpgaDevice
[
https://issues.apache.org/jira/browse/YARN-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-9268:
---
Attachment: YARN-9268-004.patch
> General improvements in FpgaDevice
> --
>
> Key: YARN-9268
> URL: https://issues.apache.org/jira/browse/YARN-9268
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9268-001.patch, YARN-9268-002.patch,
> YARN-9268-003.patch, YARN-9268-004.patch
>
>
> Need to fix the following in the class {{FpgaDevice}}:
> * It implements {{Comparable}}, but returns 0 in every case. There is no
> natural ordering among FPGA devices, perhaps "acl0" comes before "acl1", but
> this seems too forced and unnecessary.We think this class should not
> implement {{Comparable}} at all, at least not like that.
> * Stores unnecessary fields: devName, busNum, temperature, power usage. For
> one, these are never needed in the code. Secondly, temp and power usage
> changes constantly. It's pointless to store these in this POJO.
> * {{serialVersionUID}} is 1L - let's generate a number for this
> * Use {{int}} instead of {{Integer}} - don't allow nulls. If major/minor
> uniquely identifies the card, then let's demand them in the constructor and
> don't store Integers that can be null.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9267) General improvements in FpgaResourceHandlerImpl
[
https://issues.apache.org/jira/browse/YARN-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-9267:
---
Attachment: YARN-9267-007.patch
> General improvements in FpgaResourceHandlerImpl
> ---
>
> Key: YARN-9267
> URL: https://issues.apache.org/jira/browse/YARN-9267
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9267-001.patch, YARN-9267-002.patch,
> YARN-9267-003.patch, YARN-9267-004.patch, YARN-9267-005.patch,
> YARN-9267-006.patch, YARN-9267-007.patch
>
>
> Fix some problems in {{FpgaResourceHandlerImpl}}:
> * {{preStart()}} does not reconfigure card with the same IP - we see it as a
> problem. If you recompile the FPGA application, you must rename the aocx file
> because the card will not be reprogrammed. Suggestion: instead of storing
> Node<\->IPID mapping, store Node<\->IPID hash (like the SHA-256 of the
> localized file).
> * Switch to slf4j from Apache Commons Logging
> * Some unused imports
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9398) Javadoc error on FPGA related java files
[
https://issues.apache.org/jira/browse/YARN-9398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795928#comment-16795928
]
Peter Bacsko commented on YARN-9398:
Thanks [~eyang] for reporting. Will fix these soon.
> Javadoc error on FPGA related java files
>
>
> Key: YARN-9398
> URL: https://issues.apache.org/jira/browse/YARN-9398
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Peter Bacsko
>Priority: Major
>
> {code}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @param for conf
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @return
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @param for timeout
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @return
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:64:
> warning: no @return
> [ERROR] String getFpgaType();
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @return
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @throws for
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerException
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/IntelFpgaOpenclPlugin.java:156:
> error: bad HTML entity
> [ERROR]* Helper class to run aocl diagnose & determine major/minor
> numbers.
> {code}
> YARN-9266 introduced some javadoc compilation errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9398) Javadoc error on FPGA related java files
[
https://issues.apache.org/jira/browse/YARN-9398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko reassigned YARN-9398:
--
Assignee: Peter Bacsko
> Javadoc error on FPGA related java files
>
>
> Key: YARN-9398
> URL: https://issues.apache.org/jira/browse/YARN-9398
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Peter Bacsko
>Priority: Major
>
> {code}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @param for conf
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:46:
> warning: no @return
> [ERROR] boolean initPlugin(Configuration conf);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @param for timeout
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:51:
> warning: no @return
> [ERROR] boolean diagnose(int timeout);
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/AbstractFpgaVendorPlugin.java:64:
> warning: no @return
> [ERROR] String getFpgaType();
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @return
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/FpgaDiscoverer.java:119:
> warning: no @throws for
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerException
> [ERROR] public List discover()
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/fpga/IntelFpgaOpenclPlugin.java:156:
> error: bad HTML entity
> [ERROR]* Helper class to run aocl diagnose & determine major/minor
> numbers.
> {code}
> YARN-9266 introduced some javadoc compilation errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9268) General improvements in FpgaDevice
[
https://issues.apache.org/jira/browse/YARN-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795918#comment-16795918
]
Peter Bacsko commented on YARN-9268:
[~devaraj.k] this patch depends on the previous ones (the same applies to the
remaining ones). I'll upload a new one which can be tried after YARN-9267 is on
trunk, but the Jenkins build will most likely fail and need a re-trigger.
> General improvements in FpgaDevice
> --
>
> Key: YARN-9268
> URL: https://issues.apache.org/jira/browse/YARN-9268
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9268-001.patch, YARN-9268-002.patch,
> YARN-9268-003.patch
>
>
> Need to fix the following in the class {{FpgaDevice}}:
> * It implements {{Comparable}}, but returns 0 in every case. There is no
> natural ordering among FPGA devices, perhaps "acl0" comes before "acl1", but
> this seems too forced and unnecessary.We think this class should not
> implement {{Comparable}} at all, at least not like that.
> * Stores unnecessary fields: devName, busNum, temperature, power usage. For
> one, these are never needed in the code. Secondly, temp and power usage
> changes constantly. It's pointless to store these in this POJO.
> * {{serialVersionUID}} is 1L - let's generate a number for this
> * Use {{int}} instead of {{Integer}} - don't allow nulls. If major/minor
> uniquely identifies the card, then let's demand them in the constructor and
> don't store Integers that can be null.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9267) General improvements in FpgaResourceHandlerImpl
[
https://issues.apache.org/jira/browse/YARN-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795916#comment-16795916
]
Peter Bacsko commented on YARN-9267:
[~devaraj.k] yes there was a commit in the meantime which caused conflicts.
Please try patch v7.
> General improvements in FpgaResourceHandlerImpl
> ---
>
> Key: YARN-9267
> URL: https://issues.apache.org/jira/browse/YARN-9267
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9267-001.patch, YARN-9267-002.patch,
> YARN-9267-003.patch, YARN-9267-004.patch, YARN-9267-005.patch,
> YARN-9267-006.patch, YARN-9267-007.patch
>
>
> Fix some problems in {{FpgaResourceHandlerImpl}}:
> * {{preStart()}} does not reconfigure card with the same IP - we see it as a
> problem. If you recompile the FPGA application, you must rename the aocx file
> because the card will not be reprogrammed. Suggestion: instead of storing
> Node<\->IPID mapping, store Node<\->IPID hash (like the SHA-256 of the
> localized file).
> * Switch to slf4j from Apache Commons Logging
> * Some unused imports
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9080) Bucket Directories as part of ATS done accumulates
[ https://issues.apache.org/jira/browse/YARN-9080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795856#comment-16795856 ] Rakesh Shah commented on YARN-9080: --- Hello [~Prabhu Joseph] Since each time it is filtering depending upon the number of directories so, each time the call is going to namenode api, let's assume if the number directories are 1000 then that number of time it will call to the namenode api which will increase the traffic > Bucket Directories as part of ATS done accumulates > -- > > Key: YARN-9080 > URL: https://issues.apache.org/jira/browse/YARN-9080 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: 0001-YARN-9080.patch, 0002-YARN-9080.patch, > 0003-YARN-9080.patch, YARN-9080-004.patch > > > Have observed older bucket directories cluster_timestamp, bucket1 and bucket2 > as part of ATS done accumulates. The cleanLogs part of EntityLogCleaner > removes only the app directories and not the bucket directories. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8967) Change FairScheduler to use PlacementRule interface
[
https://issues.apache.org/jira/browse/YARN-8967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795771#comment-16795771
]
Hadoop QA commented on YARN-8967:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 17 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 0 new + 16 unchanged - 1 fixed = 16 total (was 17) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 338 unchanged - 67 fixed = 338 total (was 405)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 41s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 79m
23s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8967 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12962908/YARN-8967.009.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3c5616c2df58 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 568d3ab |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23744/testReport/ |
| Max. process+thread count | 870 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
[jira] [Commented] (YARN-9365) fix wrong command in TimelineServiceV2.md
[
https://issues.apache.org/jira/browse/YARN-9365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795749#comment-16795749
]
Hudson commented on YARN-9365:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16243 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16243/])
YARN-9365. Fix table creation command in TimelineServiceV2.md (rohithsharmaks:
rev c9e50c48de4ff7ef6bb89153362531833fd8da9f)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
> fix wrong command in TimelineServiceV2.md
> --
>
> Key: YARN-9365
> URL: https://issues.apache.org/jira/browse/YARN-9365
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.2.0, 3.1.1
>Reporter: Runlin Zhang
>Assignee: Runlin Zhang
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9365.patch
>
>
> In TimelineServiceV2.md 255，The step to create the timeline service schema
> does not work
>
> {noformat}
> Finally, run the schema creator tool to create the necessary tables:
> bin/hadoop
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
> -create{noformat}
>
> should be
>
> {noformat}
> The schema creation can be run on the hbase cluster which is going to store
> the timeline
> service tables. The schema creator tool requires both the
> timelineservice-hbase as well
> as the hbase-server jars. Hence, during schema creation, you need to ensure
> that the
> hbase classpath contains the yarn-timelineservice-hbase jar.
> On the hbase cluster, you can get it from hdfs since we placed it there for
> the
> coprocessor in the step above.
> ```
>hadoop fs -get
> /hbase/coprocessor/hadoop-yarn-server-timelineservice-hbase-client-${project.version}.jar
>hadoop fs -get
> /hbase/coprocessor/hadoop-yarn-server-timelineservice-${project.version}.jar
>hadoop fs -get
> /hbase/coprocessor/hadoop-yarn-server-timelineservice-hbase-common-${project.version}.jar
> /.
> ```
> Next, add it to the hbase classpath as follows:
> ```
>export
> HBASE_CLASSPATH=$HBASE_CLASSPATH:/home/yarn/hadoop-current/share/hadoop/yarn/timelineservice/hadoop-yarn-server-timelineservice-hbase-client-${project.version}.jar
>export
> HBASE_CLASSPATH=$HBASE_CLASSPATH:/home/yarn/hadoop-current/share/hadoop/yarn/timelineservice/hadoop-yarn-server-timelineservice-${project.version}.jar
>export
> HBASE_CLASSPATH=$HBASE_CLASSPATH:/home/yarn/hadoop-current/share/hadoop/yarn/timelineservice/hadoop-yarn-server-timelineservice-hbase-common-${project.version}.jar
> ```
> Finally, run the schema creator tool to create the necessary tables:
> ```
> bin/hbase
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
> -create
> ```{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
