[jira] [Updated] (YARN-5731) Preemption does not work in few corner cases when reservations are placed
[ https://issues.apache.org/jira/browse/YARN-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5731: - Attachment: YARN-5731.002.patch Rebased to proper trunk version. > Preemption does not work in few corner cases when reservations are placed > - > > Key: YARN-5731 > URL: https://issues.apache.org/jira/browse/YARN-5731 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.8.0 >Reporter: Sunil G >Assignee: Wangda Tan > Attachments: YARN-5731.001.patch, YARN-5731.002.patch, > YARN-5731-branch-2.8.001.patch > > > YARN Capacity Scheduler does not kick Preemption under below scenario. > Two queues A and B each with 50% capacity and 100% maximum capacity and user > limit factor 2. Minimum Container size is 1536MB and total cluster resource > is 40GB. Now submit the first job which needs 1536MB for AM and 9 task > containers each 4.5GB to queue A. Job will get 8 containers total (AM 1536MB > + 7 * 4.5GB = 33GB) and the cluster usage is 93.8% and the job has reserved a > container of 4.5GB. > Now when next job (1536MB for AM and 9 task containers each 4.5GB) is > submitted onto queue B. The job hangs in ACCEPTED state forever and RM > scheduler never kicks in Preemption. (RM UI Image 2 attached) > Test Case: > ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client > --queue A --executor-memory 4G --executor-cores 4 --num-executors 9 > ../lib/spark-examples*.jar 100 > After a minute.. > ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client > --queue B --executor-memory 4G --executor-cores 4 --num-executors 9 > ../lib/spark-examples*.jar 100 > Credit to: [~Prabhu Joseph] for bug investigation and troubleshooting. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5731) Preemption does not work in few corner cases when reservations are placed
[ https://issues.apache.org/jira/browse/YARN-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5731: - Attachment: YARN-5731.001.patch Attached patch for trunk, I think we should fix this corner case in trunk as well. cc: [~sunilg]. > Preemption does not work in few corner cases when reservations are placed > - > > Key: YARN-5731 > URL: https://issues.apache.org/jira/browse/YARN-5731 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.8.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-5731.001.patch, YARN-5731-branch-2.8.001.patch > > > YARN Capacity Scheduler does not kick Preemption under below scenario. > Two queues A and B each with 50% capacity and 100% maximum capacity and user > limit factor 2. Minimum Container size is 1536MB and total cluster resource > is 40GB. Now submit the first job which needs 1536MB for AM and 9 task > containers each 4.5GB to queue A. Job will get 8 containers total (AM 1536MB > + 7 * 4.5GB = 33GB) and the cluster usage is 93.8% and the job has reserved a > container of 4.5GB. > Now when next job (1536MB for AM and 9 task containers each 4.5GB) is > submitted onto queue B. The job hangs in ACCEPTED state forever and RM > scheduler never kicks in Preemption. (RM UI Image 2 attached) > Test Case: > ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client > --queue A --executor-memory 4G --executor-cores 4 --num-executors 9 > ../lib/spark-examples*.jar 100 > After a minute.. > ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client > --queue B --executor-memory 4G --executor-cores 4 --num-executors 9 > ../lib/spark-examples*.jar 100 > Credit to: [~Prabhu Joseph] for bug investigation and troubleshooting. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5731) Preemption does not work in few corner cases when reservations are placed
[ https://issues.apache.org/jira/browse/YARN-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5731: - Description: YARN Capacity Scheduler does not kick Preemption under below scenario. Two queues A and B each with 50% capacity and 100% maximum capacity and user limit factor 2. Minimum Container size is 1536MB and total cluster resource is 40GB. Now submit the first job which needs 1536MB for AM and 9 task containers each 4.5GB to queue A. Job will get 8 containers total (AM 1536MB + 7 * 4.5GB = 33GB) and the cluster usage is 93.8% and the job has reserved a container of 4.5GB. Now when next job (1536MB for AM and 9 task containers each 4.5GB) is submitted onto queue B. The job hangs in ACCEPTED state forever and RM scheduler never kicks in Preemption. (RM UI Image 2 attached) Test Case: ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --queue A --executor-memory 4G --executor-cores 4 --num-executors 9 ../lib/spark-examples*.jar 100 After a minute.. ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --queue B --executor-memory 4G --executor-cores 4 --num-executors 9 ../lib/spark-examples*.jar 100 was: Preemption doesnt kick in under below scenario. Two queues A and B each with 50% capacity and 100% maximum capacity and user limit factor 2. Job which is submitted to queueA has taken 95% of resources in cluster. Remaining 5% was also reserved by same job as demand was still higher. Now submit a small job with AM container size is lesser to above mentioned 5%. Job waits and no preemption is happening. > Preemption does not work in few corner cases when reservations are placed > - > > Key: YARN-5731 > URL: https://issues.apache.org/jira/browse/YARN-5731 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.8.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-5731-branch-2.8.001.patch > > > YARN Capacity Scheduler does not kick Preemption under below scenario. > Two queues A and B each with 50% capacity and 100% maximum capacity and user > limit factor 2. Minimum Container size is 1536MB and total cluster resource > is 40GB. Now submit the first job which needs 1536MB for AM and 9 task > containers each 4.5GB to queue A. Job will get 8 containers total (AM 1536MB > + 7 * 4.5GB = 33GB) and the cluster usage is 93.8% and the job has reserved a > container of 4.5GB. > Now when next job (1536MB for AM and 9 task containers each 4.5GB) is > submitted onto queue B. The job hangs in ACCEPTED state forever and RM > scheduler never kicks in Preemption. (RM UI Image 2 attached) > Test Case: > ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client > --queue A --executor-memory 4G --executor-cores 4 --num-executors 9 > ../lib/spark-examples*.jar 100 > After a minute.. > ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client > --queue B --executor-memory 4G --executor-cores 4 --num-executors 9 > ../lib/spark-examples*.jar 100 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5731) Preemption does not work in few corner cases when reservations are placed
[ https://issues.apache.org/jira/browse/YARN-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5731: - Description: YARN Capacity Scheduler does not kick Preemption under below scenario. Two queues A and B each with 50% capacity and 100% maximum capacity and user limit factor 2. Minimum Container size is 1536MB and total cluster resource is 40GB. Now submit the first job which needs 1536MB for AM and 9 task containers each 4.5GB to queue A. Job will get 8 containers total (AM 1536MB + 7 * 4.5GB = 33GB) and the cluster usage is 93.8% and the job has reserved a container of 4.5GB. Now when next job (1536MB for AM and 9 task containers each 4.5GB) is submitted onto queue B. The job hangs in ACCEPTED state forever and RM scheduler never kicks in Preemption. (RM UI Image 2 attached) Test Case: ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --queue A --executor-memory 4G --executor-cores 4 --num-executors 9 ../lib/spark-examples*.jar 100 After a minute.. ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --queue B --executor-memory 4G --executor-cores 4 --num-executors 9 ../lib/spark-examples*.jar 100 Credit to: [~Prabhu Joseph] for bug investigation and troubleshooting. was: YARN Capacity Scheduler does not kick Preemption under below scenario. Two queues A and B each with 50% capacity and 100% maximum capacity and user limit factor 2. Minimum Container size is 1536MB and total cluster resource is 40GB. Now submit the first job which needs 1536MB for AM and 9 task containers each 4.5GB to queue A. Job will get 8 containers total (AM 1536MB + 7 * 4.5GB = 33GB) and the cluster usage is 93.8% and the job has reserved a container of 4.5GB. Now when next job (1536MB for AM and 9 task containers each 4.5GB) is submitted onto queue B. The job hangs in ACCEPTED state forever and RM scheduler never kicks in Preemption. (RM UI Image 2 attached) Test Case: ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --queue A --executor-memory 4G --executor-cores 4 --num-executors 9 ../lib/spark-examples*.jar 100 After a minute.. ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --queue B --executor-memory 4G --executor-cores 4 --num-executors 9 ../lib/spark-examples*.jar 100 > Preemption does not work in few corner cases when reservations are placed > - > > Key: YARN-5731 > URL: https://issues.apache.org/jira/browse/YARN-5731 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.8.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-5731-branch-2.8.001.patch > > > YARN Capacity Scheduler does not kick Preemption under below scenario. > Two queues A and B each with 50% capacity and 100% maximum capacity and user > limit factor 2. Minimum Container size is 1536MB and total cluster resource > is 40GB. Now submit the first job which needs 1536MB for AM and 9 task > containers each 4.5GB to queue A. Job will get 8 containers total (AM 1536MB > + 7 * 4.5GB = 33GB) and the cluster usage is 93.8% and the job has reserved a > container of 4.5GB. > Now when next job (1536MB for AM and 9 task containers each 4.5GB) is > submitted onto queue B. The job hangs in ACCEPTED state forever and RM > scheduler never kicks in Preemption. (RM UI Image 2 attached) > Test Case: > ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client > --queue A --executor-memory 4G --executor-cores 4 --num-executors 9 > ../lib/spark-examples*.jar 100 > After a minute.. > ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client > --queue B --executor-memory 4G --executor-cores 4 --num-executors 9 > ../lib/spark-examples*.jar 100 > Credit to: [~Prabhu Joseph] for bug investigation and troubleshooting. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5731) Preemption does not work in few corner cases when reservations are placed
[ https://issues.apache.org/jira/browse/YARN-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5731: - Attachment: YARN-5731-branch-2.8.001.patch Attached patch for the fix. > Preemption does not work in few corner cases when reservations are placed > - > > Key: YARN-5731 > URL: https://issues.apache.org/jira/browse/YARN-5731 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.8.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-5731-branch-2.8.001.patch > > > Preemption doesnt kick in under below scenario. > Two queues A and B each with 50% capacity and 100% maximum capacity and user > limit factor 2. Job which is submitted to queueA has taken 95% of resources > in cluster. Remaining 5% was also reserved by same job as demand was still > higher. > Now submit a small job with AM container size is lesser to above mentioned > 5%. Job waits and no preemption is happening. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5731) Preemption does not work in few corner cases when reservations are placed
[ https://issues.apache.org/jira/browse/YARN-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5731: - Affects Version/s: (was: 2.7.3) 2.8.0 > Preemption does not work in few corner cases when reservations are placed > - > > Key: YARN-5731 > URL: https://issues.apache.org/jira/browse/YARN-5731 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.8.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-5731-branch-2.8.001.patch > > > Preemption doesnt kick in under below scenario. > Two queues A and B each with 50% capacity and 100% maximum capacity and user > limit factor 2. Job which is submitted to queueA has taken 95% of resources > in cluster. Remaining 5% was also reserved by same job as demand was still > higher. > Now submit a small job with AM container size is lesser to above mentioned > 5%. Job waits and no preemption is happening. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5731) Preemption does not work in few corner cases when reservations are placed
[ https://issues.apache.org/jira/browse/YARN-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-5731: -- Summary: Preemption does not work in few corner cases when reservations are placed (was: Preemption does not work in few corner cases where reservations are placed) > Preemption does not work in few corner cases when reservations are placed > - > > Key: YARN-5731 > URL: https://issues.apache.org/jira/browse/YARN-5731 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.7.3 >Reporter: Sunil G >Assignee: Sunil G > > Preemption doesnt kick in under below scenario. > Two queues A and B each with 50% capacity and 100% maximum capacity and user > limit factor 2. Job which is submitted to queueA has taken 95% of resources > in cluster. Remaining 5% was also reserved by same job as demand was still > higher. > Now submit a small job with AM container size is lesser to above mentioned > 5%. Job waits and no preemption is happening. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
