Loading status
I am not sure what Loading status means, followed by Running. In the application UI, I see: Executor Summary ExecutorID Worker Cores Memory State Logs 1 worker-20150202144112-hadoop-w-1.c.fi-mdd-poc.internal-3887416 83971 LOADING stdout stderr 0 worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-5868516 83971 RUNNING stdout stderr Looking at the executor hadoop-w-2, I see the status is Loading . Why different statuses, and what does that mean? Please see below for details: ID: worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685 Master URL: spark://hadoop-m:7077 Cores: 16 (16 Used) Memory: 82.0 GB (82.0 GB Used) Back to Master Running Executors (1) ExecutorID Cores State Memory Job Details Logs 0 16 LOADING 82.0 GB ID: app-20150202152154-0001 Name: Simple File Merge Application User: hadoop stdout stderr Thank you, Ami -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Loading-status-tp21468.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Loading status
Yes On Monday, February 2, 2015, Mark Hamstra m...@clearstorydata.com wrote: LOADING is just the state in which new Executors are created but before they have everything they need and are fully registered to transition to state RUNNING and begin doing actual work: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L351 https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala#L133 LOADING should be a fairly brief, transitory state. Are you seeing Executors remaining in LOADING for a significant length of time? On Mon, Feb 2, 2015 at 7:56 AM, akhandeshi ami.khande...@gmail.com javascript:_e(%7B%7D,'cvml','ami.khande...@gmail.com'); wrote: I am not sure what Loading status means, followed by Running. In the application UI, I see: Executor Summary ExecutorID Worker Cores Memory State Logs 1 worker-20150202144112-hadoop-w-1.c.fi-mdd-poc.internal-38874 16 83971 LOADING stdout stderr 0 worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685 16 83971 RUNNING stdout stderr Looking at the executor hadoop-w-2, I see the status is Loading . Why different statuses, and what does that mean? Please see below for details: ID: worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685 Master URL: spark://hadoop-m:7077 Cores: 16 (16 Used) Memory: 82.0 GB (82.0 GB Used) Back to Master Running Executors (1) ExecutorID Cores State Memory Job Details Logs 0 16 LOADING 82.0 GB ID: app-20150202152154-0001 Name: Simple File Merge Application User: hadoop stdout stderr Thank you, Ami -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Loading-status-tp21468.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org javascript:_e(%7B%7D,'cvml','user-unsubscr...@spark.apache.org'); For additional commands, e-mail: user-h...@spark.apache.org javascript:_e(%7B%7D,'cvml','user-h...@spark.apache.org');
Re: Loading status
LOADING is just the state in which new Executors are created but before they have everything they need and are fully registered to transition to state RUNNING and begin doing actual work: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L351 https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala#L133 LOADING should be a fairly brief, transitory state. Are you seeing Executors remaining in LOADING for a significant length of time? On Mon, Feb 2, 2015 at 7:56 AM, akhandeshi ami.khande...@gmail.com wrote: I am not sure what Loading status means, followed by Running. In the application UI, I see: Executor Summary ExecutorID Worker Cores Memory State Logs 1 worker-20150202144112-hadoop-w-1.c.fi-mdd-poc.internal-38874 16 83971 LOADING stdout stderr 0 worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685 16 83971 RUNNING stdout stderr Looking at the executor hadoop-w-2, I see the status is Loading . Why different statuses, and what does that mean? Please see below for details: ID: worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685 Master URL: spark://hadoop-m:7077 Cores: 16 (16 Used) Memory: 82.0 GB (82.0 GB Used) Back to Master Running Executors (1) ExecutorID Cores State Memory Job Details Logs 0 16 LOADING 82.0 GB ID: app-20150202152154-0001 Name: Simple File Merge Application User: hadoop stdout stderr Thank you, Ami -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Loading-status-tp21468.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Loading status
It seems sort of Listener UI error! I say this because, I see the status in the executor web UI to be loading, but the application UI, for the same executor the status is Running! I have also seen the reverse behavior where the application UI indicates a particular executor as loading, but the executor UI page indicates it is running! There are no error messages in the logs that I have been able to spot! Also, logs seems to indicate it is doing the work.. As far as reproducibility goes, it is hard, due to its inconsistent behavior observed...perhaps, again indicating some listener issue... will update as I find any further information... thank you both for help! On Monday, February 2, 2015, Mark Hamstra m...@clearstorydata.com wrote: Yes, if the Master is unable to register the Executor and transition it to RUNNING, then the Executor will stay in LOADING state, so this can be caused by problems in the Master or the Master-Executor communication. On Mon, Feb 2, 2015 at 9:24 AM, Tushar Sharma tushars...@gmail.com javascript:_e(%7B%7D,'cvml','tushars...@gmail.com'); wrote: Yes curious indeed. Usually if such status persists it leads to executor failure after a couple of tries. We also noticed similar behavior but the logs clearly mentioned the lack of some akka resource as the reason. I am sure the logs also holds the key for this particular problem. On Mon, Feb 2, 2015 at 10:49 PM, Mark Hamstra m...@clearstorydata.com javascript:_e(%7B%7D,'cvml','m...@clearstorydata.com'); wrote: Curious. I guess the first question is whether we've got some sort of Listener/UI error so that the UI is not accurately reflecting the Executor's actual state, or whether the LOADING Executor really is spending a considerable length of time in this I'm in the process of being created, but not yet doing anything useful state. If you can figure out a little more of what is going on or how to reproduce this state, please do file a JIRA. On Mon, Feb 2, 2015 at 8:28 AM, Ami Khandeshi ami.khande...@gmail.com javascript:_e(%7B%7D,'cvml','ami.khande...@gmail.com'); wrote: Yes On Monday, February 2, 2015, Mark Hamstra m...@clearstorydata.com javascript:_e(%7B%7D,'cvml','m...@clearstorydata.com'); wrote: LOADING is just the state in which new Executors are created but before they have everything they need and are fully registered to transition to state RUNNING and begin doing actual work: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L351 https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala#L133 LOADING should be a fairly brief, transitory state. Are you seeing Executors remaining in LOADING for a significant length of time? On Mon, Feb 2, 2015 at 7:56 AM, akhandeshi ami.khande...@gmail.com wrote: I am not sure what Loading status means, followed by Running. In the application UI, I see: Executor Summary ExecutorID Worker Cores Memory State Logs 1 worker-20150202144112-hadoop-w-1.c.fi-mdd-poc.internal-38874 16 83971 LOADING stdout stderr 0 worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685 16 83971 RUNNING stdout stderr Looking at the executor hadoop-w-2, I see the status is Loading . Why different statuses, and what does that mean? Please see below for details: ID: worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685 Master URL: spark://hadoop-m:7077 Cores: 16 (16 Used) Memory: 82.0 GB (82.0 GB Used) Back to Master Running Executors (1) ExecutorID Cores State Memory Job Details Logs 0 16 LOADING 82.0 GB ID: app-20150202152154-0001 Name: Simple File Merge Application User: hadoop stdout stderr Thank you, Ami -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Loading-status-tp21468.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Loading status
Yes, if the Master is unable to register the Executor and transition it to RUNNING, then the Executor will stay in LOADING state, so this can be caused by problems in the Master or the Master-Executor communication. On Mon, Feb 2, 2015 at 9:24 AM, Tushar Sharma tushars...@gmail.com wrote: Yes curious indeed. Usually if such status persists it leads to executor failure after a couple of tries. We also noticed similar behavior but the logs clearly mentioned the lack of some akka resource as the reason. I am sure the logs also holds the key for this particular problem. On Mon, Feb 2, 2015 at 10:49 PM, Mark Hamstra m...@clearstorydata.com wrote: Curious. I guess the first question is whether we've got some sort of Listener/UI error so that the UI is not accurately reflecting the Executor's actual state, or whether the LOADING Executor really is spending a considerable length of time in this I'm in the process of being created, but not yet doing anything useful state. If you can figure out a little more of what is going on or how to reproduce this state, please do file a JIRA. On Mon, Feb 2, 2015 at 8:28 AM, Ami Khandeshi ami.khande...@gmail.com wrote: Yes On Monday, February 2, 2015, Mark Hamstra m...@clearstorydata.com wrote: LOADING is just the state in which new Executors are created but before they have everything they need and are fully registered to transition to state RUNNING and begin doing actual work: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L351 https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala#L133 LOADING should be a fairly brief, transitory state. Are you seeing Executors remaining in LOADING for a significant length of time? On Mon, Feb 2, 2015 at 7:56 AM, akhandeshi ami.khande...@gmail.com wrote: I am not sure what Loading status means, followed by Running. In the application UI, I see: Executor Summary ExecutorID Worker Cores Memory State Logs 1 worker-20150202144112-hadoop-w-1.c.fi-mdd-poc.internal-38874 16 83971 LOADING stdout stderr 0 worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685 16 83971 RUNNING stdout stderr Looking at the executor hadoop-w-2, I see the status is Loading . Why different statuses, and what does that mean? Please see below for details: ID: worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685 Master URL: spark://hadoop-m:7077 Cores: 16 (16 Used) Memory: 82.0 GB (82.0 GB Used) Back to Master Running Executors (1) ExecutorID Cores State Memory Job Details Logs 0 16 LOADING 82.0 GB ID: app-20150202152154-0001 Name: Simple File Merge Application User: hadoop stdout stderr Thank you, Ami -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Loading-status-tp21468.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Loading status
Curious. I guess the first question is whether we've got some sort of Listener/UI error so that the UI is not accurately reflecting the Executor's actual state, or whether the LOADING Executor really is spending a considerable length of time in this I'm in the process of being created, but not yet doing anything useful state. If you can figure out a little more of what is going on or how to reproduce this state, please do file a JIRA. On Mon, Feb 2, 2015 at 8:28 AM, Ami Khandeshi ami.khande...@gmail.com wrote: Yes On Monday, February 2, 2015, Mark Hamstra m...@clearstorydata.com wrote: LOADING is just the state in which new Executors are created but before they have everything they need and are fully registered to transition to state RUNNING and begin doing actual work: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L351 https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala#L133 LOADING should be a fairly brief, transitory state. Are you seeing Executors remaining in LOADING for a significant length of time? On Mon, Feb 2, 2015 at 7:56 AM, akhandeshi ami.khande...@gmail.com wrote: I am not sure what Loading status means, followed by Running. In the application UI, I see: Executor Summary ExecutorID Worker Cores Memory State Logs 1 worker-20150202144112-hadoop-w-1.c.fi-mdd-poc.internal-38874 16 83971 LOADING stdout stderr 0 worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685 16 83971 RUNNING stdout stderr Looking at the executor hadoop-w-2, I see the status is Loading . Why different statuses, and what does that mean? Please see below for details: ID: worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685 Master URL: spark://hadoop-m:7077 Cores: 16 (16 Used) Memory: 82.0 GB (82.0 GB Used) Back to Master Running Executors (1) ExecutorID Cores State Memory Job Details Logs 0 16 LOADING 82.0 GB ID: app-20150202152154-0001 Name: Simple File Merge Application User: hadoop stdout stderr Thank you, Ami -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Loading-status-tp21468.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
