[jira] [Resolved] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-1963. --- Resolution: Fixed Closing this issue as all sub-items are done. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: 0001-YARN-1963-prototype.patch, YARN Application > Priorities Design _02.pdf, YARN Application Priorities Design.pdf, YARN > Application Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2099) Preemption in fair scheduler should consider app priorities
[ https://issues.apache.org/jira/browse/YARN-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2099: -- Issue Type: Bug (was: Sub-task) Parent: (was: YARN-1963) > Preemption in fair scheduler should consider app priorities > --- > > Key: YARN-2099 > URL: https://issues.apache.org/jira/browse/YARN-2099 > Project: Hadoop YARN > Issue Type: Bug > Components: api, resourcemanager >Affects Versions: 2.5.0 >Reporter: Ashwin Shankar >Assignee: Wei Yan > Attachments: YARN-2099.patch > > > Fair scheduler should take app priorities into account while > preempting containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2098) App priority support in Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223854#comment-16223854 ] Sunil G commented on YARN-2098: --- linking to YARN-1963 > App priority support in Fair Scheduler > -- > > Key: YARN-2098 > URL: https://issues.apache.org/jira/browse/YARN-2098 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.5.0 >Reporter: Ashwin Shankar >Assignee: Wei Yan > Attachments: YARN-2098.patch, YARN-2098.patch > > > This jira is created for supporting app priorities in fair scheduler. > AppSchedulable hard codes priority of apps to 1,we should > change this to get priority from ApplicationSubmissionContext. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2098) App priority support in Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2098: -- Issue Type: Bug (was: Sub-task) Parent: (was: YARN-1963) > App priority support in Fair Scheduler > -- > > Key: YARN-2098 > URL: https://issues.apache.org/jira/browse/YARN-2098 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.5.0 >Reporter: Ashwin Shankar >Assignee: Wei Yan > Attachments: YARN-2098.patch, YARN-2098.patch > > > This jira is created for supporting app priorities in fair scheduler. > AppSchedulable hard codes priority of apps to 1,we should > change this to get priority from ApplicationSubmissionContext. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7414) FairScheduler#getAppWeight() should be moved into FSAppAttempt#getWeight()
[
https://issues.apache.org/jira/browse/YARN-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Soumabrata Chakraborty reassigned YARN-7414:
Assignee: Soumabrata Chakraborty
> FairScheduler#getAppWeight() should be moved into FSAppAttempt#getWeight()
> --
>
> Key: YARN-7414
> URL: https://issues.apache.org/jira/browse/YARN-7414
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-beta1
>Reporter: Daniel Templeton
>Assignee: Soumabrata Chakraborty
>Priority: Minor
> Labels: newbie
>
> It's illogical that {{FSAppAttempt}} defers to {{FairScheduler}} for its own
> weight, especially when {{FairScheduler}} has to call back to
> {{FSAppAttempt}} to get the details to return a value. Instead,
> {{FSAppAttempt}} should do the work and call out to {{FairScheduler}} to get
> the details it needs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations
[
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223840#comment-16223840
]
Arun Suresh commented on YARN-4511:
---
Thanks for updating the patch [~haibochen]. I am able to follow this a bit
better.
Some Comments:
- In SchedulerNode, you are doing an if check inside
{{guaranteedContainerResourceAllocated}}, but you do not do the same in
{{opportunisticContainerResourceAllocated}}. Is there a case where the
'resource' argument might be null in the former method call (since that is only
case when the {{containerResourceAllocated}} method can return false)?
{code}
if (containerResourceAllocated(resource, allocatedResourceGuaranteed)) {
Resources.subtractFrom(unallocatedResource, resource);
}
{code}
- I see a similar pattern as above in {{guaranteedContainerResourceReleased}}
and {{opportunisticContainerResourceReleased}}
- Thank you for splitting numContainers into numGuaranteedContainers and
numAllocatedContainers, but then in the SchedulerNodeReport, shouldn't this.num
= numOpp + numGuaranteed ?
Everything else looks fine to me.
> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch,
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch,
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch,
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch,
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch,
> YARN-4511-YARN-1011.09.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7119) yarn rmadmin -updateNodeResource should be updated for resource types
[
https://issues.apache.org/jira/browse/YARN-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223839#comment-16223839
]
Sunil G commented on YARN-7119:
---
Thanks [~maniraj...@gmail.com]
Few comments
# {{CapacitySchedulerConfiguration.getUnits}} could be used instead of
replicating same in ResourceUtils.
# {{-updateNodeResource}} is not related to resource-profiles feature
enable/disable boolean flag. We have resource types and resource profiles as
separate features and config is only for profiles. YARN by default will support
multiple resource types if resoruce-types.xml is supplied. So pls update the
help in RMAdminCLI and remove the profile flag check.
# checkMandatoryResources in RMAdminCLI should be reused or make it as common
as possible to ResourceUtils
# RMAdminCLI#createResource is more or less parsing the resource types from
incoming string and creating a resource out of it. So lets say it as
parseCommandAndCreateResource
> yarn rmadmin -updateNodeResource should be updated for resource types
> -
>
> Key: YARN-7119
> URL: https://issues.apache.org/jira/browse/YARN-7119
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Manikandan R
> Attachments: YARN-7119.001.patch, YARN-7119.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[
https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223838#comment-16223838
]
Sunil G commented on YARN-7159:
---
Thanks [~maniraj...@gmail.com].
Some more comments
# In {{ResourceUtils.getNodeResourceInformation}} also, its better to normalize
the resource to what is already loaded as per *resouorce-types.xml* (if this
conf is available, usually when u run RM and NM in same machine, both files
could be there)
# Lets add more tests to TestResourcePBImpl in cases like a smaller unit is
converted to a higher one and vice versa.
# Another scenario is when one NM loads resource in GB (memory) as per
node-resource.xml and verify whther NM is registered with units MB. This test
case is better to add.
> Normalize unit of resource objects in RM and avoid to do unit conversion in
> critical path
> -
>
> Key: YARN-7159
> URL: https://issues.apache.org/jira/browse/YARN-7159
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Manikandan R
>Priority: Critical
> Attachments: YARN-7159.001.patch, YARN-7159.002.patch,
> YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch
>
>
> Currently resource conversion could happen in critical code path when
> different unit is specified by client. This could impact performance and
> throughput of RM a lot. We should do unit normalization when resource passed
> to RM and avoid expensive unit conversion every time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6984) DominantResourceCalculator.isAnyMajorResourceZero() should test all resources
[
https://issues.apache.org/jira/browse/YARN-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223826#comment-16223826
]
Sunil G commented on YARN-6984:
---
Thanks [~templedf]! Appreciate the same.
> DominantResourceCalculator.isAnyMajorResourceZero() should test all resources
> -
>
> Key: YARN-6984
> URL: https://issues.apache.org/jira/browse/YARN-6984
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Sunil G
> Fix For: 3.1.0
>
> Attachments: YARN-6984.001.patch, YARN-6984.002.patch
>
>
> The method currently tests only memory and CPU. It looks to me like it
> should test all resources, i.e. it should do what {{isInvalidDivisor()}} does
> and should, in fact, replace that method. [~sunilg], since you wrote the
> method originally, can you comment on what its intended semantics are?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations
[
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223824#comment-16223824
]
Wangda Tan commented on YARN-4511:
--
[~haibo.chen],
Haven't reviewed all details of the patch, but the latest patch looks much
cleaner than previous one. I took a closer look at SchedulerNode.
Few comments:
- {{assert}} in the main code (Such as SchedulerNode) will be removed at
runtime. Do you want to throw exception instead?
- Moving following statements:
{code}
// do not update allocated containers until the resources of the
// container are released because we need to check if we need
// to update resourceAllocatedPendingLaunch in case the container
// has not been launched on the node.
allocatedContainers.remove(containerId);
numGuaranteedContainers--;
{code}
To {{guaranteedContainerResourceReleased}}? And same for the
{{opportunisticContainerResourceReleased}}
- Duplicated isDebugEnabled:
{code}
if (LOG.isDebugEnabled()) {
if (LOG.isDebugEnabled()) {
{code}
Please proceed if you get a +1 from other committer. (Since I haven't reviewed
all other logics)
+ [~asuresh] in case he wants to take a look.
> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch,
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch,
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch,
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch,
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch,
> YARN-4511-YARN-1011.09.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7224) Support GPU isolation for docker container
[
https://issues.apache.org/jira/browse/YARN-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223823#comment-16223823
]
Sunil G commented on YARN-7224:
---
Committing shortly.
> Support GPU isolation for docker container
> --
>
> Key: YARN-7224
> URL: https://issues.apache.org/jira/browse/YARN-7224
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-7224.001.patch, YARN-7224.002-wip.patch,
> YARN-7224.003.patch, YARN-7224.004.patch, YARN-7224.005.patch,
> YARN-7224.006.patch, YARN-7224.007.patch, YARN-7224.008.patch,
> YARN-7224.009.patch
>
>
> This patch is to address issues when docker container is being used:
> 1. GPU driver and nvidia libraries: If GPU drivers and NV libraries are
> pre-packaged inside docker image, it could conflict to driver and
> nvidia-libraries installed on Host OS. An alternative solution is to detect
> Host OS's installed drivers and devices, mount it when launch docker
> container. Please refer to \[1\] for more details.
> 2. Image detection:
> From \[2\], the challenge is:
> bq. Mounting user-level driver libraries and device files clobbers the
> environment of the container, it should be done only when the container is
> running a GPU application. The challenge here is to determine if a given
> image will be using the GPU or not. We should also prevent launching
> containers based on a Docker image that is incompatible with the host NVIDIA
> driver version, you can find more details on this wiki page.
> 3. GPU isolation.
> *Proposed solution*:
> a. Use nvidia-docker-plugin \[3\] to address issue #1, this is the same
> solution used by K8S \[4\]. issue #2 could be addressed in a separate JIRA.
> We won't ship nvidia-docker-plugin with out releases and we require cluster
> admin to preinstall nvidia-docker-plugin to use GPU+docker support on YARN.
> "nvidia-docker" is a wrapper of docker binary which can address #3 as well,
> however "nvidia-docker" doesn't provide same semantics of docker, and it
> needs to setup additional environments such as PATH/LD_LIBRARY_PATH to use
> it. To avoid introducing additional issues, we plan to use
> nvidia-docker-plugin + docker binary approach.
> b. To address GPU driver and nvidia libraries, we uses nvidia-docker-plugin
> \[3\] to create a volume which includes GPU-related libraries and mount it
> when docker container being launched. Changes include:
> - Instead of using {{volume-driver}}, this patch added {{docker volume
> create}} command to c-e and NM Java side. The reason is {{volume-driver}} can
> only use single volume driver for each launched docker container.
> - Updated {{c-e}} and Java side, if a mounted volume is a named volume in
> docker, skip checking file existence. (Named-volume still need to be added to
> permitted list of container-executor.cfg).
> c. To address isolation issue:
> We found that, cgroup + docker doesn't work under newer docker version which
> uses {{runc}} as default runtime. Setting {{--cgroup-parent}} to a cgroup
> which include any {{devices.deny}} causes docker container cannot be launched.
> Instead this patch passes allowed GPU devices via {{--device}} to docker
> launch command.
> References:
> \[1\] https://github.com/NVIDIA/nvidia-docker/wiki/NVIDIA-driver
> \[2\] https://github.com/NVIDIA/nvidia-docker/wiki/Image-inspection
> \[3\] https://github.com/NVIDIA/nvidia-docker/wiki/nvidia-docker-plugin
> \[4\] https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7410) Cleanup FixedValueResource to avoid dependency to ResourceUtils
[ https://issues.apache.org/jira/browse/YARN-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223818#comment-16223818 ] Wangda Tan commented on YARN-7410: -- Ran the perf test (TestCapacitySchedulerPerf) several more times. It looks like previous reported performance is not correct: Performance numbers of before/after the patch: 2 Resource types: ~55-56K 3 Resource types: ~43-44K 4 Resource types: ~41-42K 5 Resource types: ~39-40K [~sunilg] are you able to reproduce this experiment? If yes, I think we're good in terms of performance. > Cleanup FixedValueResource to avoid dependency to ResourceUtils > --- > > Key: YARN-7410 > URL: https://issues.apache.org/jira/browse/YARN-7410 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.0 >Reporter: Sunil G >Assignee: Wangda Tan > Attachments: YARN-7410.001.patch, YARN-7410.002.patch > > > Currently FixedValue Resource constants has some dependencies to > ResourceUtils. This jira will help to cleanup this dependencies. > Thanks [~leftnoteasy] for finding the same. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223804#comment-16223804
]
Wangda Tan commented on YARN-6594:
--
Thanks [~kkaranasos] for updating the JIRA.
+1, will commit the patch next Monday.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594-YARN-6592.002.patch, YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7415) RM Rest API Submit application documentation not easy to understand
Atul Kulkarni created YARN-7415: --- Summary: RM Rest API Submit application documentation not easy to understand Key: YARN-7415 URL: https://issues.apache.org/jira/browse/YARN-7415 Project: Hadoop YARN Issue Type: Bug Components: api, docs, documentation Affects Versions: 2.7.3 Reporter: Atul Kulkarni Priority: Minor This specifically pertains to the "Cluster Applications API(Submit Application)" documentation This line - “Please note that in order to submit an app, you must have an authentication filter setup for the HTTP interface. The functionality requires that a username is set in the HttpServletRequest. If no filter is setup, the response will be an “UNAUTHORIZED” response.” is NOT very helpful in conveying to the user what needs to happen on the client side or on the REST API side. Specifically, 1. "Authentication filter setup for the HTTP interface" - * Now does this mean that one needs Kerberos enabled on the cluster? * If not, what kind of HTTP authentication filter needs to be setup on the httpd or on the client side? A few wikipedia or other links to do this would go long way in increasing adoption of the RM REST API. 2. "The functionality requires that a username is set in the HttpServletRequest" - * HttpServletRequest is a Java class - does the REST API mandate integrations use Java? * If not what would be an equivalent of that in normal HTTP REST Parlance? This will help people understand what to do when using Python etc. I frustrated myself with this documentation over the last few days and very many "tutorials" that claim to make it work on production clusters and finally reached this conclusion that - if this documentation can be improved, it would save everyone a LOT of time and effort. I am happy to help fix this - I have no experience contributing to apache projects and may make mistakes but I am willing to learn. I would need answers to the above questions to be able to do anything with this. I still have not figured out how to run the job without hdfs complaining about user permissions. Which I am guessing are related. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7380) Check findbugs in timeline service branch-2
[ https://issues.apache.org/jira/browse/YARN-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-7380: - Target Version/s: 2.9.0 Description: Some findbugs warnings have been noticed in the branch-2 nightly builds. I haven't investigated them yet but filing to confirm/fix if possible. I recollect some known findbugs issues with the webservices function calls with number of parameters which we could not fix. was: Some findbugs warnings have been noticed in the branch-2 nightly builds. I haven't investigated them yet but filing to confirm/fix if possible. I recollect some known findbugs issues with the webservices function calls with number of parameters which we could not fix. > Check findbugs in timeline service branch-2 > --- > > Key: YARN-7380 > URL: https://issues.apache.org/jira/browse/YARN-7380 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Attachments: YARN-7380-branch-2.0001.patch > > > Some findbugs warnings have been noticed in the branch-2 nightly builds. I > haven't investigated them yet but filing to confirm/fix if possible. > I recollect some known findbugs issues with the webservices function calls > with number of parameters which we could not fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7380) Check findbugs in timeline service branch-2
[ https://issues.apache.org/jira/browse/YARN-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-7380: - Attachment: YARN-7380-branch-2.0001.patch Upload v0001 > Check findbugs in timeline service branch-2 > --- > > Key: YARN-7380 > URL: https://issues.apache.org/jira/browse/YARN-7380 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Attachments: YARN-7380-branch-2.0001.patch > > > Some findbugs warnings have been noticed in the branch-2 nightly builds. I > haven't investigated them yet but filing to confirm/fix if possible. > I recollect some known findbugs issues with the webservices function calls > with number of parameters which we could not fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7380) Check findbugs in timeline service branch-2
[
https://issues.apache.org/jira/browse/YARN-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223689#comment-16223689
]
Vrushali C commented on YARN-7380:
--
The findbugs warning that I see is
{code}
hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/src/main/javahadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/src/test/java
{code}
> Check findbugs in timeline service branch-2
> ---
>
> Key: YARN-7380
> URL: https://issues.apache.org/jira/browse/YARN-7380
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>
> Some findbugs warnings have been noticed in the branch-2 nightly builds. I
> haven't investigated them yet but filing to confirm/fix if possible.
> I recollect some known findbugs issues with the webservices function calls
> with number of parameters which we could not fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7380) Check findbugs in timeline service branch-2
[
https://issues.apache.org/jira/browse/YARN-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223689#comment-16223689
]
Vrushali C edited comment on YARN-7380 at 10/28/17 6:16 PM:
The findbugs warning that I see is in hadoop-yarn-server-timelineservice-hbase
module:
{code}
hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/src/main/javahadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/src/test/java
{code}
was (Author: vrushalic):
The findbugs warning that I see is
{code}
hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/src/main/javahadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/src/test/java
{code}
> Check findbugs in timeline service branch-2
> ---
>
> Key: YARN-7380
> URL: https://issues.apache.org/jira/browse/YARN-7380
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>
> Some findbugs warnings have been noticed in the branch-2 nightly builds. I
> haven't investigated them yet but filing to confirm/fix if possible.
> I recollect some known findbugs issues with the webservices function calls
> with number of parameters which we could not fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: (was: YARN-7276-branch-2.005.patch)
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276.000.patch, YARN-7276.001.patch,
> YARN-7276.002.patch, YARN-7276.003.patch, YARN-7276.004.patch,
> YARN-7276.005.patch, YARN-7276.006.patch, YARN-7276.007.patch,
> YARN-7276.009.patch, YARN-7276.010.patch, YARN-7276.011.patch,
> YARN-7276.012.patch, YARN-7276.013.patch, YARN-7276.014.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.005.patch
Not sure the warnings are correct (i.e., checkstyle), I'm uploading
[^YARN-7276-branch-2.005.patch] again to see if it goes away.
One error that I've seen before is about the unchecked conversion with:
{code}
public Map getParameterMap() {
return hsr.getParameterMap();
}
{code}
Trunk (Java 8) didn't complain but with branch-2 is happening once in a while.
Not sure how to fix it.
I found the following:
https://stackoverflow.com/questions/5317443/java-unchecked-conversion
However, it doesn't look like a clean solution.
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276.000.patch, YARN-7276.001.patch,
> YARN-7276.002.patch, YARN-7276.003.patch, YARN-7276.004.patch,
> YARN-7276.005.patch, YARN-7276.006.patch, YARN-7276.007.patch,
> YARN-7276.009.patch, YARN-7276.010.patch, YARN-7276.011.patch,
> YARN-7276.012.patch, YARN-7276.013.patch, YARN-7276.014.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7397) Reduce lock contention in FairScheduler#getAppWeight()
[
https://issues.apache.org/jira/browse/YARN-7397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223635#comment-16223635
]
Hudson commented on YARN-7397:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13156 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13156/])
YARN-7397. Reduce lock contention in FairScheduler#getAppWeight() (templedf:
rev e62bbbca7adafa0e050212e99c41c95a844700ff)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
> Reduce lock contention in FairScheduler#getAppWeight()
> --
>
> Key: YARN-7397
> URL: https://issues.apache.org/jira/browse/YARN-7397
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Fix For: 3.1.0
>
> Attachments: YARN-7397.001.patch
>
>
> In profiling the fair scheduler, a large amount of time is spent waiting to
> get the lock in {{FairScheduler.getAppWeight()}}, when the lock isn't
> actually needed. This patch reduces the scope of the lock to eliminate that
> contention.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7397) Reduce lock contention in FairScheduler#getAppWeight()
[
https://issues.apache.org/jira/browse/YARN-7397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-7397:
---
Affects Version/s: (was: 3.0.0-beta1)
3.1.0
> Reduce lock contention in FairScheduler#getAppWeight()
> --
>
> Key: YARN-7397
> URL: https://issues.apache.org/jira/browse/YARN-7397
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Fix For: 3.1.0
>
> Attachments: YARN-7397.001.patch
>
>
> In profiling the fair scheduler, a large amount of time is spent waiting to
> get the lock in {{FairScheduler.getAppWeight()}}, when the lock isn't
> actually needed. This patch reduces the scope of the lock to eliminate that
> contention.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7414) FairScheduler#getAppWeight() should be moved into FSAppAttempt#getWeight()
Daniel Templeton created YARN-7414:
--
Summary: FairScheduler#getAppWeight() should be moved into
FSAppAttempt#getWeight()
Key: YARN-7414
URL: https://issues.apache.org/jira/browse/YARN-7414
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Affects Versions: 3.0.0-beta1
Reporter: Daniel Templeton
Priority: Minor
It's illogical that {{FSAppAttempt}} defers to {{FairScheduler}} for its own
weight, especially when {{FairScheduler}} has to call back to {{FSAppAttempt}}
to get the details to return a value. Instead, {{FSAppAttempt}} should do the
work and call out to {{FairScheduler}} to get the details it needs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7397) Reduce lock contention in FairScheduler#getAppWeight()
[
https://issues.apache.org/jira/browse/YARN-7397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223624#comment-16223624
]
Daniel Templeton commented on YARN-7397:
It's not exactly straightforward to untangle that because of
{{sizeBasedWeight}}. I'll file another JIRA for it.
> Reduce lock contention in FairScheduler#getAppWeight()
> --
>
> Key: YARN-7397
> URL: https://issues.apache.org/jira/browse/YARN-7397
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-beta1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-7397.001.patch
>
>
> In profiling the fair scheduler, a large amount of time is spent waiting to
> get the lock in {{FairScheduler.getAppWeight()}}, when the lock isn't
> actually needed. This patch reduces the scope of the lock to eliminate that
> contention.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7369) Improve the resource types docs
[ https://issues.apache.org/jira/browse/YARN-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223621#comment-16223621 ] Sunil G commented on YARN-7369: --- bq.but they won't be new anymore. Yes. The existing statement to put them in same dir should be enough. bq.Would it help to add a statement in the config section that the preferred method is to place the new properties in the new config files? In config section, all properties were mentioned with description. In next section, structure of same config props are given. So to avoid confusion for admin, second one could be tagged as a sample resource-types.xml to add resources like resource1/2 etc. > Improve the resource types docs > --- > > Key: YARN-7369 > URL: https://issues.apache.org/jira/browse/YARN-7369 > Project: Hadoop YARN > Issue Type: Sub-task > Components: docs >Affects Versions: 3.1.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: YARN-7369.001.patch, YARN-7369.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7369) Improve the resource types docs
[ https://issues.apache.org/jira/browse/YARN-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7369: --- Attachment: YARN-7369.002.patch See if this works better for you. > Improve the resource types docs > --- > > Key: YARN-7369 > URL: https://issues.apache.org/jira/browse/YARN-7369 > Project: Hadoop YARN > Issue Type: Sub-task > Components: docs >Affects Versions: 3.1.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: YARN-7369.001.patch, YARN-7369.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7369) Improve the resource types docs
[
https://issues.apache.org/jira/browse/YARN-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223560#comment-16223560
]
Daniel Templeton edited comment on YARN-7369 at 10/28/17 3:34 PM:
--
Thanks for the review, [~sunilg]!
# There's this line: {quote}Please note that the `resource-types.xml` and
`node-resources.xml` files also need to be placed in same configuration
directory as `yarn-site.xml` if they are used.{quote} It doesn't say that the
files are new, but I was trying not to say that. By Hadoop 4.0, this doc will
still be there, but they won't be new anymore.
# Yeah, the config and model sections both reference the properties, but the
duplication is OK in my opinion. In the config section they're just enumerated,
whereas the model section explains them in text. Basically the same content,
but I don't see an issue in making it crystal clear. Would it help to add a
statement in the config section that the preferred method is to place the new
properties in the new config files?
# I'll add that.
was (Author: templedf):
Thanks for the review, @sunil!
# There's this line: {quote}Please note that the `resource-types.xml` and
`node-resources.xml` files also need to be placed in same configuration
directory as `yarn-site.xml` if they are used.{quote} It doesn't say that the
files are new, but I was trying not to say that. By Hadoop 4.0, this doc will
still be there, but they won't be new anymore.
# Yeah, the config and model sections both reference the properties, but the
duplication is OK in my opinion. In the config section they're just enumerated,
whereas the model section explains them in text. Basically the same content,
but I don't see an issue in making it crystal clear. Would it help to add a
statement in the config section that the preferred method is to place the new
properties in the new config files?
# I'll add that.
> Improve the resource types docs
> ---
>
> Key: YARN-7369
> URL: https://issues.apache.org/jira/browse/YARN-7369
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: docs
>Affects Versions: 3.1.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-7369.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7369) Improve the resource types docs
[
https://issues.apache.org/jira/browse/YARN-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223560#comment-16223560
]
Daniel Templeton commented on YARN-7369:
Thanks for the review, @sunil!
# There's this line: {quote}Please note that the `resource-types.xml` and
`node-resources.xml` files also need to be placed in same configuration
directory as `yarn-site.xml` if they are used.{quote} It doesn't say that the
files are new, but I was trying not to say that. By Hadoop 4.0, this doc will
still be there, but they won't be new anymore.
# Yeah, the config and model sections both reference the properties, but the
duplication is OK in my opinion. In the config section they're just enumerated,
whereas the model section explains them in text. Basically the same content,
but I don't see an issue in making it crystal clear. Would it help to add a
statement in the config section that the preferred method is to place the new
properties in the new config files?
# I'll add that.
> Improve the resource types docs
> ---
>
> Key: YARN-7369
> URL: https://issues.apache.org/jira/browse/YARN-7369
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: docs
>Affects Versions: 3.1.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-7369.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[
https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223522#comment-16223522
]
Hadoop QA commented on YARN-7159:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
29s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 26s{color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
23s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.api.TestResourcePBImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7159 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12894561/YARN-7159.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 1ceb078c6dba 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9c5c687 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_131 |
| mvninstall |
https://builds.apache.org/job/PreCommit-YARN-Build/18223/artifact/out/branch-mvninstall-root.txt
|
| findbugs | v3.1.0-RC1
[jira] [Commented] (YARN-7119) yarn rmadmin -updateNodeResource should be updated for resource types
[
https://issues.apache.org/jira/browse/YARN-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223493#comment-16223493
]
Hadoop QA commented on YARN-7119:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
48s{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 5m
12s{color} | {color:red} hadoop-yarn in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
24s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 54s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 15 new + 102 unchanged - 0 fixed = 117 total (was 102) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 48s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
21s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 34s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
25s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api |
| | Boxing/unboxing to parse a primitive
org.apache.hadoop.yarn.util.resource.ResourceUtils.getValue(String) At
ResourceUtils.java:org.apache.hadoop.yarn.util.resource.ResourceUtils.getValue(String)
At ResourceUtils.java:[line 436] |
| Failed junit tests | hadoop.yarn.client.cli.TestRMAdminCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7119 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12894560/YARN-7119.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
[jira] [Updated] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-7159: --- Attachment: YARN-7159.005.patch > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Manikandan R >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223466#comment-16223466 ] Manikandan R commented on YARN-7159: Rebased. Attached new patch. > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Manikandan R >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R reassigned YARN-7159: -- Assignee: Manikandan R (was: Wangda Tan) > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Manikandan R >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7119) yarn rmadmin -updateNodeResource should be updated for resource types
[
https://issues.apache.org/jira/browse/YARN-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223450#comment-16223450
]
Manikandan R commented on YARN-7119:
[~sunilg] [~leftnoteasy] Thanks for working on YARN-7307.
Had an offline discussion with [~sunilg] about YARN-7307 and changes required
for this JIRA based on YARN-7307.
Summary is, Modified the whole approach to start using
{{ResourceUtils#getResourceTypes()}} rather than depending on
{{NodeManagerHardwareUtils.java}} to create resource and pass it to the
remaining flow - {{ResourceManagerAdministrationProtocol}}. To keep the
validation simple, {{RMAdminCLI#checkMandatoryResources}} expects mandatory
resources from users. Also this method is slightly similar to mandatory
resources checks in {{NodeManagerHardwareUtils#getNodeResources}}, so planned
to move this piece to common place, possibly in {{hadoop-yarn-common}} based on
the feasibility in separate jira.
> yarn rmadmin -updateNodeResource should be updated for resource types
> -
>
> Key: YARN-7119
> URL: https://issues.apache.org/jira/browse/YARN-7119
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Manikandan R
> Attachments: YARN-7119.001.patch, YARN-7119.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7119) yarn rmadmin -updateNodeResource should be updated for resource types
[ https://issues.apache.org/jira/browse/YARN-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-7119: --- Attachment: YARN-7119.002.patch > yarn rmadmin -updateNodeResource should be updated for resource types > - > > Key: YARN-7119 > URL: https://issues.apache.org/jira/browse/YARN-7119 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Manikandan R > Attachments: YARN-7119.001.patch, YARN-7119.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[
https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223433#comment-16223433
]
Hadoop QA commented on YARN-7159:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-7159 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7159 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12894555/YARN-7159.004.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18221/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Normalize unit of resource objects in RM and avoid to do unit conversion in
> critical path
> -
>
> Key: YARN-7159
> URL: https://issues.apache.org/jira/browse/YARN-7159
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Critical
> Attachments: YARN-7159.001.patch, YARN-7159.002.patch,
> YARN-7159.003.patch, YARN-7159.004.patch
>
>
> Currently resource conversion could happen in critical code path when
> different unit is specified by client. This could impact performance and
> throughput of RM a lot. We should do unit normalization when resource passed
> to RM and avoid expensive unit conversion every time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-7159: --- Attachment: YARN-7159.004.patch > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223431#comment-16223431 ] Manikandan R commented on YARN-7159: Fixed jnit failures, whitespace & checkstyle issues. > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223344#comment-16223344
]
Eric Yang commented on YARN-7197:
-
Btw, docker doesn't support double mount like:
{code}
docker run -it -v /mnt/hdfs/user:/home -v /tmp/empty:/home/yarn
{code}
If {{/mnt/hdfs/user}} is already mounted, then yarn will show up. We have to
trust that yarn directory permission is setup correctly. Bind mount a
directory with blank directory will not solve any problem to hide /home/yarn
directory. Filesystem ACL is the only safe guard. It's not possible to make a
subdirectory disappear.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223306#comment-16223306
]
Eric Yang commented on YARN-7197:
-
[~jlowe] I agree with everything you said that the current implementation is a
moot point besides top level directories. The real security problem with mount
was solved in YARN-4266, where enforcing effective group will ensure the
processes inside the container doesn't get more privileges than the user was
allowed. The intend of black list is supposed to add additional safe guard to
prevent mistakes that can result in jail break. If we go by what you said,
then it would be {{/mnt/hdfs/user/yarn}} in black list, it automatically gets
replaced with {{/dev/null}}.
If someone make an attack like:
{code}
docker run -v /mnt/hdfs/user/yarn:/tmp/yarn centos:latest bash
{code}
The resulting command looks like:
{code}
docker run -v /dev/null:/tmp/yarn centos:latest bash
{code}
Inside the container looks like:
{code}
[root@6d6588764109 /]# ls -l /tmp
total 4
-rwx-- 1 root root 836 Sep 11 15:53 ks-script-q6TWGF
crw-rw-rw- 1 root root 1, 3 Oct 28 07:37 yarn
-rw--- 1 root root0 Sep 11 15:51 yum.log
{code}
Everything goes no where, and no clean up. Thoughts?
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6927) Add support for individual resource types requests in MapReduce
[
https://issues.apache.org/jira/browse/YARN-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223299#comment-16223299
]
Hadoop QA commented on YARN-6927:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 38s{color} | {color:orange} root: The patch generated 6 new + 879 unchanged
- 3 fixed = 885 total (was 882) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 7s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
53s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
56s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}123m 26s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
42s{color} | {color:red} The patch generated 4 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}220m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.mapred.pipes.TestPipeApplication |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-6927 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12894506/YARN-6927.008.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0103e91684f6
[jira] [Commented] (YARN-7412) test_docker_util.test_check_mount_permitted() is failing
[ https://issues.apache.org/jira/browse/YARN-7412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223292#comment-16223292 ] Eric Yang commented on YARN-7412: - Different distros seem to link /usr/bin to /bin or vis-versa. Maybe it is safer to choose a file out side of /usr/bin or /bin? How about /etc/profile? > test_docker_util.test_check_mount_permitted() is failing > > > Key: YARN-7412 > URL: https://issues.apache.org/jira/browse/YARN-7412 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: Haibo Chen >Assignee: Eric Badger >Priority: Critical > Attachments: YARN-7412.001.patch > > > Test output > classname="TestDockerUtil"> >message="/home/haibochen/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:444 > Expected: itr-second Which is: 1To be equal to: > ret Which is: 0for inp > ut /usr/bin/touch" > type=""> > > classname="TestDockerUtil"> >message="/home/haibochen/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:462 > Expected: expected[i] Which is: > /usr/bin/touchTo be equal to: ptr[i] > Which is: /bin/touch" > type=""> > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7369) Improve the resource types docs
[
https://issues.apache.org/jira/browse/YARN-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223288#comment-16223288
]
Sunil G commented on YARN-7369:
---
Thank you [~templedf] for helping in this. Few quick comments.
# I think its better to clearly mention that {{`resource-types.xml`}}
{{`node-resources.xml`}} are new files introduced and one has to configure
these to get new resource types in to YARN.
# In {{YARN Resource Model}} sections, all properties are expanded further. Is
it a duplication to {{Configuration}} section. And also if we are planning to
indicate some config items for better clarity, its better to add that these
configs has to be in {{`resource-types.xml`}}.
# As a note, it may be better to mention that one can use resource types alone
with out enabling profiles config also.
> Improve the resource types docs
> ---
>
> Key: YARN-7369
> URL: https://issues.apache.org/jira/browse/YARN-7369
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: docs
>Affects Versions: 3.1.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-7369.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7299) Fix TestDistributedScheduler
[
https://issues.apache.org/jira/browse/YARN-7299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223283#comment-16223283
]
Hudson commented on YARN-7299:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13155 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13155/])
YARN-7299. Fix TestDistributedScheduler. (asuresh) (arun suresh: rev
9c5c68745ed18883ce8bdbdca379025b23f17f60)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/scheduler/TestDistributedScheduler.java
> Fix TestDistributedScheduler
>
>
> Key: YARN-7299
> URL: https://issues.apache.org/jira/browse/YARN-7299
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: Arun Suresh
> Fix For: 2.9.0, 3.0.0
>
> Attachments: YARN-7299.001.patch
>
>
> TestDistributedScheduler has been failing consistently in trunk:
> {noformat}
> Running
> org.apache.hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.75 sec <<<
> FAILURE! - in
> org.apache.hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler
> testDistributedScheduler(org.apache.hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler)
> Time elapsed: 0.67 sec <<< FAILURE!
> java.lang.AssertionError: expected:<4> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler.testDistributedScheduler(TestDistributedScheduler.java:118)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7178) Add documentation for Container Update API
[ https://issues.apache.org/jira/browse/YARN-7178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223275#comment-16223275 ] Hudson commented on YARN-7178: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13154 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13154/]) YARN-7178. Add documentation for Container Update API. (asuresh) (arun suresh: rev 24f8c5cce3adb88e5af84b3e48c4c447ac91e6d3) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md > Add documentation for Container Update API > -- > > Key: YARN-7178 > URL: https://issues.apache.org/jira/browse/YARN-7178 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh >Priority: Blocker > Fix For: 2.9.0, 3.0.0 > > Attachments: YARN-7178.001.patch, YARN-7178.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7299) Fix TestDistributedScheduler
[
https://issues.apache.org/jira/browse/YARN-7299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-7299:
--
Summary: Fix TestDistributedScheduler (was: TestDistributedScheduler is
failing)
> Fix TestDistributedScheduler
>
>
> Key: YARN-7299
> URL: https://issues.apache.org/jira/browse/YARN-7299
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: Arun Suresh
> Attachments: YARN-7299.001.patch
>
>
> TestDistributedScheduler has been failing consistently in trunk:
> {noformat}
> Running
> org.apache.hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.75 sec <<<
> FAILURE! - in
> org.apache.hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler
> testDistributedScheduler(org.apache.hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler)
> Time elapsed: 0.67 sec <<< FAILURE!
> java.lang.AssertionError: expected:<4> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler.testDistributedScheduler(TestDistributedScheduler.java:118)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
