[jira] [Created] (YARN-2722) Disable SSLv3 (POODLEbleed vulnerability) in YARN shuffle

2014-10-21 Thread Wei Yan (JIRA)
Wei Yan created YARN-2722: - Summary: Disable SSLv3 (POODLEbleed vulnerability) in YARN shuffle Key: YARN-2722 URL: https://issues.apache.org/jira/browse/YARN-2722 Project: Hadoop YARN Issue Type:

[jira] [Updated] (YARN-2722) Disable SSLv3 (POODLEbleed vulnerability) in YARN shuffle

2014-10-22 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2722: -- Attachment: YARN-2722-1.patch This patch creates a whilelist {TLSv1.2, TLSv1.1, TLSv1} for the SSLFactory. Have

[jira] [Commented] (YARN-2722) Disable SSLv3 (POODLEbleed vulnerability) in YARN shuffle

2014-10-23 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181588#comment-14181588 ] Wei Yan commented on YARN-2722: --- Thanks, [~schu]. You're right, we shouldn't add TLSv1.2. And

[jira] [Commented] (YARN-2722) Disable SSLv3 (POODLEbleed vulnerability) in YARN shuffle

2014-10-23 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181677#comment-14181677 ] Wei Yan commented on YARN-2722: --- Hi, [~schu]. Discussed with Robert offline, and we also need

[jira] [Updated] (YARN-2722) Disable SSLv3 (POODLEbleed vulnerability) in YARN shuffle

2014-10-23 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2722: -- Attachment: YARN-2722-2.patch Disable SSLv3 (POODLEbleed vulnerability) in YARN shuffle

[jira] [Commented] (YARN-1021) Yarn Scheduler Load Simulator

2014-03-02 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13917634#comment-13917634 ] Wei Yan commented on YARN-1021: --- [~qzhang90]. Check the resource simulate.info.html.template.

[jira] [Updated] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions

2014-03-24 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-596: - Attachment: YARN-596.patch [~sandyr], [~kkambatl]. Could you guys take a look of this patch? The premption rules

[jira] [Updated] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions

2014-03-24 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-596: - Attachment: YARN-596.patch In fair scheduler, intra-application container priorities affect inter-application

[jira] [Updated] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions

2014-03-24 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-596: - Attachment: YARN-596.patch In fair scheduler, intra-application container priorities affect inter-application

[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions

2014-03-28 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13951126#comment-13951126 ] Wei Yan commented on YARN-596: -- Thought that approach before. But I forget why give up. I'll

[jira] [Updated] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions

2014-03-28 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-596: - Attachment: YARN-596.patch Update a new patch without cloneQueueApps(). Instead, add preemptedResources in the

[jira] [Updated] (YARN-1726) ResourceSchedulerWrapper failed due to the AbstractYarnScheduler introduced in YARN-1041

2014-03-31 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1726: -- Attachment: YARN-1726.patch A new patch: (1) Fix the problem because of the AbstractYarnScheduler. (2) Add

[jira] [Updated] (YARN-1726) ResourceSchedulerWrapper failed due to the AbstractYarnScheduler introduced in YARN-1041

2014-04-01 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1726: -- Attachment: YARN-1726.patch ResourceSchedulerWrapper failed due to the AbstractYarnScheduler introduced in

[jira] [Updated] (YARN-1726) ResourceSchedulerWrapper failed due to the AbstractYarnScheduler introduced in YARN-1041

2014-04-01 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1726: -- Attachment: YARN-1726.patch ResourceSchedulerWrapper failed due to the AbstractYarnScheduler introduced in

[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions

2014-04-07 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962488#comment-13962488 ] Wei Yan commented on YARN-596: -- {quote}For preemptContainerPreCheck, it would be good to take

[jira] [Commented] (YARN-1021) Yarn Scheduler Load Simulator

2014-04-28 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13982963#comment-13982963 ] Wei Yan commented on YARN-1021: --- [~haiouman], Sorry for late reply. Great that your problem

[jira] [Commented] (YARN-2073) FairScheduler starts preempting resources even with free resources on the cluster

2014-05-20 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004175#comment-14004175 ] Wei Yan commented on YARN-2073: --- [~kasha], if we move preemption-related test code to a

[jira] [Assigned] (YARN-1913) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

2014-05-21 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan reassigned YARN-1913: - Assignee: Wei Yan (was: Karthik Kambatla) With Fair Scheduler, cluster can logjam when all resources

[jira] [Updated] (YARN-1913) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

2014-05-21 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1913: -- Attachment: YARN-1913.patch init patch for review. Add queueMaxAMShare configuration for each queue. And update

[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions

2014-05-22 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006248#comment-14006248 ] Wei Yan commented on YARN-596: -- Hey, [~sandyr], sorry for the late reply. Still confuse here.

[jira] [Updated] (YARN-1913) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

2014-05-22 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1913: -- Attachment: YARN-1913.patch With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions

2014-05-22 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006615#comment-14006615 ] Wei Yan commented on YARN-596: -- yes, we can check the queue's policy in the preCheck function.

[jira] [Updated] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions

2014-05-23 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-596: - Attachment: YARN-596.patch Update a new patch to fix Sandy's suggestions. In fair scheduler, intra-application

[jira] [Updated] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions

2014-05-23 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-596: - Attachment: YARN-596.patch In fair scheduler, intra-application container priorities affect inter-application

[jira] [Commented] (YARN-2099) Preemption in fair scheduler should consider app priorities

2014-05-23 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007789#comment-14007789 ] Wei Yan commented on YARN-2099: --- Hi, Ashwin. We have discussed a little bit about the

[jira] [Commented] (YARN-2099) Preemption in fair scheduler should consider app priorities

2014-05-23 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007844#comment-14007844 ] Wei Yan commented on YARN-2099: --- Add a PriorityPolicy besides the existing fifo/fair/drf

[jira] [Updated] (YARN-1913) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

2014-05-23 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1913: -- Attachment: YARN-1913.patch With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

[jira] [Created] (YARN-2106) TestFairScheduler in trunk is failing

2014-05-27 Thread Wei Yan (JIRA)
Wei Yan created YARN-2106: - Summary: TestFairScheduler in trunk is failing Key: YARN-2106 URL: https://issues.apache.org/jira/browse/YARN-2106 Project: Hadoop YARN Issue Type: Bug

[jira] [Commented] (YARN-1021) Yarn Scheduler Load Simulator

2014-05-27 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010234#comment-14010234 ] Wei Yan commented on YARN-1021: --- [~cristiana.voicu], the SLS directly supports rumen traces.

[jira] [Commented] (YARN-2099) Preemption in fair scheduler should consider app priorities

2014-05-27 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010251#comment-14010251 ] Wei Yan commented on YARN-2099: --- Hey, [~ashwinshankar77], Are you working on this one? If

[jira] [Assigned] (YARN-2099) Preemption in fair scheduler should consider app priorities

2014-05-27 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan reassigned YARN-2099: - Assignee: Wei Yan Preemption in fair scheduler should consider app priorities

[jira] [Updated] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions

2014-05-27 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-596: - Attachment: YARN-596.patch Update a new patch after YARN-2105 is in. In fair scheduler, intra-application

[jira] [Updated] (YARN-1913) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

2014-05-27 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1913: -- Attachment: YARN-1913.patch With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

[jira] [Updated] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions

2014-05-28 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-596: - Attachment: YARN-596.patch Thanks, Sandy. Upload a new patch to fix your comments. In fair scheduler,

[jira] [Updated] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions

2014-05-28 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-596: - Attachment: YARN-596.patch In fair scheduler, intra-application container priorities affect inter-application

[jira] [Assigned] (YARN-2098) App priority support in Fair Scheduler

2014-05-28 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan reassigned YARN-2098: - Assignee: Wei Yan App priority support in Fair Scheduler --

[jira] [Updated] (YARN-2098) App priority support in Fair Scheduler

2014-05-28 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2098: -- Attachment: YARN-2098.patch App priority support in Fair Scheduler --

[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions

2014-05-28 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14011814#comment-14011814 ] Wei Yan commented on YARN-596: -- Thanks, [~ashwinshankar77]. I'll update a patch quickly. In

[jira] [Updated] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions

2014-05-28 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-596: - Attachment: YARN-596.patch In fair scheduler, intra-application container priorities affect inter-application

[jira] [Updated] (YARN-2098) App priority support in Fair Scheduler

2014-05-29 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2098: -- Attachment: YARN-2098.patch App priority support in Fair Scheduler --

[jira] [Assigned] (YARN-596) Use scheduling policies throughout the queue hierarchy to decide which containers to preempt

2014-05-29 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan reassigned YARN-596: Assignee: Wei Yan (was: Sandy Ryza) Use scheduling policies throughout the queue hierarchy to decide which

[jira] [Updated] (YARN-1913) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

2014-05-29 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1913: -- Labels: (was: easyfix) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

[jira] [Updated] (YARN-1913) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

2014-05-29 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1913: -- Attachment: YARN-1913.patch Thanks, Sandy. Upload a new patch using the accurate way. With Fair Scheduler,

[jira] [Updated] (YARN-2099) Preemption in fair scheduler should consider app priorities

2014-05-30 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2099: -- Attachment: YARN-2099.patch Upload an initial patch to capture which we discussed. Need to add testcases once

[jira] [Commented] (YARN-1913) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

2014-05-30 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014308#comment-14014308 ] Wei Yan commented on YARN-1913: --- Thanks, Sandy. Will update a patch. With Fair Scheduler,

[jira] [Updated] (YARN-1913) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

2014-05-30 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1913: -- Attachment: YARN-1913.patch With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

[jira] [Commented] (YARN-1913) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

2014-05-30 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014422#comment-14014422 ] Wei Yan commented on YARN-1913: --- Update a new patch to fix Sandy's comments.

[jira] [Updated] (YARN-1913) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

2014-06-02 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1913: -- Attachment: YARN-1913.patch Thanks, Sandy. Upload a new patch, move the AM resource usage check to

[jira] [Updated] (YARN-1913) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

2014-06-02 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1913: -- Attachment: YARN-1913.patch With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

[jira] [Commented] (YARN-1913) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

2014-06-02 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14015536#comment-14015536 ] Wei Yan commented on YARN-1913: --- Thanks, Sandy. One problem may exist if we use

[jira] [Updated] (YARN-1913) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

2014-06-02 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1913: -- Attachment: YARN-1913.patch Update a patch. Use getLiveContainer().size() and unManagedAM to detect the AM

[jira] [Updated] (YARN-1913) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

2014-06-02 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1913: -- Attachment: YARN-1913.patch Thanks, Sandy. Fixed that problem. With Fair Scheduler, cluster can logjam when

[jira] [Updated] (YARN-1913) With Fair Scheduler, cluster can logjam when all resources are consumed by AMs

2014-06-02 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1913: -- Attachment: YARN-1913.patch New patch to fix the test errors. With Fair Scheduler, cluster can logjam when

[jira] [Created] (YARN-2126) The FSLeafQueue.amResourceUsage shouldn't be updated when an Application removed before it runs AM

2014-06-05 Thread Wei Yan (JIRA)
Wei Yan created YARN-2126: - Summary: The FSLeafQueue.amResourceUsage shouldn't be updated when an Application removed before it runs AM Key: YARN-2126 URL: https://issues.apache.org/jira/browse/YARN-2126

[jira] [Updated] (YARN-2126) The FSLeafQueue.amResourceUsage shouldn't be updated when an Application removed before it runs AM

2014-06-05 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2126: -- Description: When an application is removed, the FSLeafQueue updates its amResourceUsage. {code} if

[jira] [Created] (YARN-2128) SchedulerApplicationAttempt's amResource should be normalized instead of fetching from ApplicationSubmissionContext directly

2014-06-05 Thread Wei Yan (JIRA)
Wei Yan created YARN-2128: - Summary: SchedulerApplicationAttempt's amResource should be normalized instead of fetching from ApplicationSubmissionContext directly Key: YARN-2128 URL:

[jira] [Updated] (YARN-2128) SchedulerApplicationAttempt's amResource should be normalized instead of fetching from ApplicationSubmissionContext directly

2014-06-05 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2128: -- Attachment: YARN-2128.patch SchedulerApplicationAttempt's amResource should be normalized instead of

[jira] [Resolved] (YARN-2126) The FSLeafQueue.amResourceUsage shouldn't be updated when an Application removed before it runs AM

2014-06-06 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan resolved YARN-2126. --- Resolution: Duplicate Close it as duplicate. Will fixed by YARN-2128. The FSLeafQueue.amResourceUsage

[jira] [Updated] (YARN-2128) SchedulerApplicationAttempt's amResource should be normalized instead of fetching from ApplicationSubmissionContext directly

2014-06-06 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2128: -- Attachment: YARN-2128.patch Update a new patch combing YARN-2126. SchedulerApplicationAttempt's amResource

[jira] [Updated] (YARN-2128) SchedulerApplicationAttempt's amResource should be normalized instead of fetching from ApplicationSubmissionContext directly

2014-06-06 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2128: -- Description: 1. The amResource should be normalized instead of fetching from ApplicationSubmissionContext

[jira] [Created] (YARN-2139) Add support for disk IO isolation/scheduling for containers

2014-06-09 Thread Wei Yan (JIRA)
Wei Yan created YARN-2139: - Summary: Add support for disk IO isolation/scheduling for containers Key: YARN-2139 URL: https://issues.apache.org/jira/browse/YARN-2139 Project: Hadoop YARN Issue Type:

[jira] [Created] (YARN-2140) Add support for network IO isolation/scheduling for containers

2014-06-09 Thread Wei Yan (JIRA)
Wei Yan created YARN-2140: - Summary: Add support for network IO isolation/scheduling for containers Key: YARN-2140 URL: https://issues.apache.org/jira/browse/YARN-2140 Project: Hadoop YARN Issue

[jira] [Commented] (YARN-2140) Add support for network IO isolation/scheduling for containers

2014-06-10 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026674#comment-14026674 ] Wei Yan commented on YARN-2140: --- Thanks, [~bikassaha]. Yes, I'm working on that part. Add

[jira] [Commented] (YARN-2139) Add support for disk IO isolation/scheduling for containers

2014-06-10 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026935#comment-14026935 ] Wei Yan commented on YARN-2139: --- For the disk IO, we can use cgroup to limit the IO usage for

[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit

2014-06-10 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026967#comment-14026967 ] Wei Yan commented on YARN-2083: --- [~tianyi], here are some comments. 1. The tab size should be

[jira] [Updated] (YARN-2126) The FSLeafQueue.amResourceUsage shouldn't be updated when an Application removed before it runs AM

2014-06-10 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2126: -- Assignee: (was: Wei Yan) The FSLeafQueue.amResourceUsage shouldn't be updated when an Application

[jira] [Commented] (YARN-2145) FairScheduler can assign more resources to queue with MaxShare set.

2014-06-11 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14027813#comment-14027813 ] Wei Yan commented on YARN-2145: --- duplicate with YARN-2083? FairScheduler can assign more

[jira] [Commented] (YARN-2073) Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free

2014-06-11 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14028256#comment-14028256 ] Wei Yan commented on YARN-2073: --- should be a mistake here. need to use allocated resource.

[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit

2014-06-12 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14029273#comment-14029273 ] Wei Yan commented on YARN-2083: --- Thanks, [~tianyi]. Here are more comments. Could we move the

[jira] [Created] (YARN-2155) FairScheduler: Incorrect check when trigger a preemption

2014-06-12 Thread Wei Yan (JIRA)
Wei Yan created YARN-2155: - Summary: FairScheduler: Incorrect check when trigger a preemption Key: YARN-2155 URL: https://issues.apache.org/jira/browse/YARN-2155 Project: Hadoop YARN Issue Type: Bug

[jira] [Updated] (YARN-2155) FairScheduler: Incorrect check when trigger a preemption

2014-06-12 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2155: -- Description: {code} private boolean shouldAttemptPreemption() { if (preemptionEnabled) { return

[jira] [Updated] (YARN-2155) FairScheduler: Incorrect check when trigger a preemption

2014-06-12 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2155: -- Description: {code} private boolean shouldAttemptPreemption() { if (preemptionEnabled) { return

[jira] [Updated] (YARN-2155) FairScheduler: Incorrect check when trigger a preemption

2014-06-12 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2155: -- Attachment: YARN-2155.patch FairScheduler: Incorrect check when trigger a preemption

[jira] [Commented] (YARN-2140) Add support for network IO isolation/scheduling for containers

2014-06-12 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14030271#comment-14030271 ] Wei Yan commented on YARN-2140: --- [~haosd...@gmail.com], [~beckham007], net_cls can be used to

[jira] [Commented] (YARN-2140) Add support for network IO isolation/scheduling for containers

2014-06-13 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14030750#comment-14030750 ] Wei Yan commented on YARN-2140: --- Thanks for the comments, [~revans2]. Add support for

[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit

2014-06-18 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14035879#comment-14035879 ] Wei Yan commented on YARN-2083: --- Hey, [~tianyi], the patch looks good to me. In fair

[jira] [Created] (YARN-2194) Add Cgroup support for RedHat 7

2014-06-23 Thread Wei Yan (JIRA)
Wei Yan created YARN-2194: - Summary: Add Cgroup support for RedHat 7 Key: YARN-2194 URL: https://issues.apache.org/jira/browse/YARN-2194 Project: Hadoop YARN Issue Type: Improvement

[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7

2014-06-23 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041263#comment-14041263 ] Wei Yan commented on YARN-2194: --- SystemdNspawnContainerExectuor is a good idea. Just add one

[jira] [Created] (YARN-2195) Clean a piece of code in ResourceRequest

2014-06-23 Thread Wei Yan (JIRA)
Wei Yan created YARN-2195: - Summary: Clean a piece of code in ResourceRequest Key: YARN-2195 URL: https://issues.apache.org/jira/browse/YARN-2195 Project: Hadoop YARN Issue Type: Improvement

[jira] [Updated] (YARN-2195) Clean a piece of code in ResourceRequest

2014-06-23 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2195: -- Attachment: YARN-2195.patch Clean a piece of code in ResourceRequest

[jira] [Commented] (YARN-2195) Clean a piece of code in ResourceRequest

2014-06-24 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14042286#comment-14042286 ] Wei Yan commented on YARN-2195: --- Thanks, [~devaraj.k]. Clean a piece of code in

[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7

2014-06-26 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14045101#comment-14045101 ] Wei Yan commented on YARN-2194: --- [~tucu00], you mean just detect whether the system is RedHat

[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7

2014-06-26 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14045110#comment-14045110 ] Wei Yan commented on YARN-2194: --- oh, yes, adding a new LCEResourceHandler to handle the

[jira] [Updated] (YARN-2139) Add support for disk IO isolation/scheduling for containers

2014-07-01 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2139: -- Attachment: Disk_IO_Scheduling_Design_1.pdf Attach a design draft. Add support for disk IO

[jira] [Commented] (YARN-2139) Add support for disk IO isolation/scheduling for containers

2014-07-02 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050301#comment-14050301 ] Wei Yan commented on YARN-2139: --- Thanks for the comments, [~ste...@apache.org]. For you

[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling

2014-07-05 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053039#comment-14053039 ] Wei Yan commented on YARN-2252: --- Thanks for the catch, [~rdsr]. For the node perference,

[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling

2014-07-06 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053157#comment-14053157 ] Wei Yan commented on YARN-2252: --- Yes, that makes sense. We can only use the old scheduler,

[jira] [Assigned] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling

2014-07-06 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan reassigned YARN-2252: - Assignee: Wei Yan Intermittent failure for testcase TestFairScheduler.testContinuousScheduling

[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling

2014-07-06 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053168#comment-14053168 ] Wei Yan commented on YARN-2252: --- Look the code again. I remembered why I created a new FS, as

[jira] [Updated] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling

2014-07-07 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2252: -- Assignee: (was: Wei Yan) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling

[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling

2014-07-07 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053790#comment-14053790 ] Wei Yan commented on YARN-2252: --- Thanks, [~rdsr]. I'll take a look later. Intermittent

[jira] [Commented] (YARN-2273) NPE in ContinuousScheduling Thread crippled RM after DN flap

2014-07-10 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057930#comment-14057930 ] Wei Yan commented on YARN-2273: --- Thanks for the catch, [~skeltoac]. A quick guess is that

[jira] [Commented] (YARN-2273) NPE in ContinuousScheduling Thread crippled RM after DN flap

2014-07-11 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14058989#comment-14058989 ] Wei Yan commented on YARN-2273: --- I run a testcase locally. Disable the continuousScheduling,

[jira] [Updated] (YARN-2273) NPE in ContinuousScheduling Thread crippled RM after DN flap

2014-07-11 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2273: -- Attachment: YARN-2273.patch NPE in ContinuousScheduling Thread crippled RM after DN flap

[jira] [Commented] (YARN-2273) NPE in ContinuousScheduling Thread crippled RM after DN flap

2014-07-11 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059096#comment-14059096 ] Wei Yan commented on YARN-2273: --- Thanks, [~ozawa]. IMO, as the nodeIdList makes a copy from

[jira] [Commented] (YARN-2273) NPE in ContinuousScheduling Thread crippled RM after DN flap

2014-07-11 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059145#comment-14059145 ] Wei Yan commented on YARN-2273: --- Oh, I saw your point. Yes, move the copy operation to the

[jira] [Updated] (YARN-2273) NPE in ContinuousScheduling Thread crippled RM after DN flap

2014-07-11 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2273: -- Attachment: YARN-2273.patch NPE in ContinuousScheduling Thread crippled RM after DN flap

[jira] [Commented] (YARN-2273) NPE in ContinuousScheduling Thread crippled RM after DN flap

2014-07-11 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059252#comment-14059252 ] Wei Yan commented on YARN-2273: --- Thanks, [~ozawa]. Update a new patch. NPE in

[jira] [Commented] (YARN-1010) FairScheduler: decouple container scheduling from nodemanager heartbeats

2013-09-30 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782437#comment-13782437 ] Wei Yan commented on YARN-1010: --- Updates in the patch. (1) The {{FairScheduler}} launches a

[jira] [Updated] (YARN-1010) FairScheduler: decouple container scheduling from nodemanager heartbeats

2013-10-01 Thread Wei Yan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-1010: -- Attachment: YARN-1010.patch FairScheduler: decouple container scheduling from nodemanager heartbeats

<    1   2   3   4   5   >