Re: How to shut down Flink Web Dashboard in detached Yarn session?
You could also use `jsp` or `ps` to check that no TaskExecutor and StandaloneJobClusterEntrypoint is running. If there are no such processes, then there should not be a Flink cluster running locally. Cheers, Till On Wed, Jan 2, 2019 at 6:31 PM Sai Inampudi wrote: > Hey Till, > > If it is running on a standalone Flink cluster, wouldn't running > stop-cluster.sh work? > > When I run stop-cluster.sh, I get back: > No taskexecutor daemon to stop on host . > No standalonesession daemon to stop on host . > > So I assumed that meant that it is not running on a standalone cluster > > On 2019/01/02 14:13:52, Till Rohrmann wrote: > > Hi Sai, > > > > could you check that the dashboard you are seeing is really running on > Yarn > > and not a standalone Flink cluster which you have running locally? > > > > Cheers, > > Till > > > > On Mon, Dec 31, 2018 at 7:40 PM Sai Inampudi > wrote: > > > > > Hey Gary, thanks for reaching out. > > > > > > Executing "yarn application -list" does not return my flink cluster so > I > > > assume like in my initial post that the application must be > terminated. My > > > config when I ran the job did not have log aggregation enabled and that > > > might be why when I try to look at the logs via "yarn logs > -applicationId > > > ", I get back nothing: > > > (e.g. Unable to get ApplicationState. Attempting to fetch logs > > > directly from the filesystem. > > > /tmp/logs/si022833/logs/application_1545041832015_73428 does not > > > exist.) > > > > > > > > > (My previous reply was formatted incorrectly so I am replying back with > > > proper formatting. Apologies for the mistake) > > > > > > > > > On 2018/12/31 18:13:05, Sai Inampudi wrote: > > > > > > > > > > > > On 2018/12/31 10:53:58, Gary Yao wrote: > > > > > Hi, > > > > > > > > > > You can use the YARN client to list all applications on your YARN > > > cluster: > > > > > > > > > > yarn application -list > > > > > > > > > > If this does not show any running applications, the Flink cluster > must > > > have > > > > > somehow terminated. If you have YARN's log aggregation enabled, you > > > should > > > > > be > > > > > able to view the Flink logs by running: > > > > > > > > > > yarn logs -applicationId > > > > > > > > > > Best, > > > > > Gary > > > > > > > > > > On Fri, Dec 28, 2018 at 9:42 PM Sai Inampudi < > sai.inamp...@gmail.com> > > > wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > I recently attempted to create a Flink cluster on YARN by > executing > > > the > > > > > > following: > > > > > > ~/flink-1.5.4/bin/yarn-session.sh -n 5 -tm 2048 -s 4 -d -nm > > > flink_yarn > > > > > > > > > > > > The resulting command was not completely successful but it did > end up > > > > > > creating a Apache Flink Dashboard with 1 Task Manager, 1 Task > Slot, > > > and 1 > > > > > > Job Manager. > > > > > > > > > > > > When I look at my Yarn Resource Manager, I don't see my > application > > > > > > running. CLI calls for the application id also returned nothing. > > > > > > > > > > > > I would like to kill the existing web dashboard as well as the > other > > > > > > lingering task manager/job manager so that I can try recreating > the > > > yarn > > > > > > session successfully. > > > > > > > > > > > > Has anyone encountered this before and has any suggestion? I > looked > > > > > > through documentation [1] which says to stop a yarn session, you > > > will want > > > > > > to use the YARN utilities (yarn application -kill ) to > stop > > > the YARN > > > > > > session. However, the application id in my logs is not found in > the > > > > > > Resource Manager so it seems to already have been killed (due to > the > > > > > > original yarn session command not properly executing?). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/yarn_setup.html#detached-yarn-session > > > > > > > > > > > Hey Gary, thanks for reaching out. > > > > Executing "yarn application -list" does not return my flink cluster > so I > > > assume like in my initial post that the application must be > terminated. My > > > config when I ran the job did not have log aggregation enabled and that > > > might be why when I try to look at the logs via "yarn logs > -applicationId > > > ", I get back nothing (e.g. Unable to get > ApplicationState. > > > Attempting to fetch logs directly from the filesystem. > > > > /tmp/logs/si022833/logs/application_1545041832015_73428 does not > exist.) > > > > > > > > > >
Re: How to shut down Flink Web Dashboard in detached Yarn session?
Hey Till, If it is running on a standalone Flink cluster, wouldn't running stop-cluster.sh work? When I run stop-cluster.sh, I get back: No taskexecutor daemon to stop on host . No standalonesession daemon to stop on host . So I assumed that meant that it is not running on a standalone cluster On 2019/01/02 14:13:52, Till Rohrmann wrote: > Hi Sai, > > could you check that the dashboard you are seeing is really running on Yarn > and not a standalone Flink cluster which you have running locally? > > Cheers, > Till > > On Mon, Dec 31, 2018 at 7:40 PM Sai Inampudi wrote: > > > Hey Gary, thanks for reaching out. > > > > Executing "yarn application -list" does not return my flink cluster so I > > assume like in my initial post that the application must be terminated. My > > config when I ran the job did not have log aggregation enabled and that > > might be why when I try to look at the logs via "yarn logs -applicationId > > ", I get back nothing: > > (e.g. Unable to get ApplicationState. Attempting to fetch logs > > directly from the filesystem. > > /tmp/logs/si022833/logs/application_1545041832015_73428 does not > > exist.) > > > > > > (My previous reply was formatted incorrectly so I am replying back with > > proper formatting. Apologies for the mistake) > > > > > > On 2018/12/31 18:13:05, Sai Inampudi wrote: > > > > > > > > > On 2018/12/31 10:53:58, Gary Yao wrote: > > > > Hi, > > > > > > > > You can use the YARN client to list all applications on your YARN > > cluster: > > > > > > > > yarn application -list > > > > > > > > If this does not show any running applications, the Flink cluster must > > have > > > > somehow terminated. If you have YARN's log aggregation enabled, you > > should > > > > be > > > > able to view the Flink logs by running: > > > > > > > > yarn logs -applicationId > > > > > > > > Best, > > > > Gary > > > > > > > > On Fri, Dec 28, 2018 at 9:42 PM Sai Inampudi > > wrote: > > > > > > > > > Hi everyone, > > > > > > > > > > I recently attempted to create a Flink cluster on YARN by executing > > the > > > > > following: > > > > > ~/flink-1.5.4/bin/yarn-session.sh -n 5 -tm 2048 -s 4 -d -nm > > flink_yarn > > > > > > > > > > The resulting command was not completely successful but it did end up > > > > > creating a Apache Flink Dashboard with 1 Task Manager, 1 Task Slot, > > and 1 > > > > > Job Manager. > > > > > > > > > > When I look at my Yarn Resource Manager, I don't see my application > > > > > running. CLI calls for the application id also returned nothing. > > > > > > > > > > I would like to kill the existing web dashboard as well as the other > > > > > lingering task manager/job manager so that I can try recreating the > > yarn > > > > > session successfully. > > > > > > > > > > Has anyone encountered this before and has any suggestion? I looked > > > > > through documentation [1] which says to stop a yarn session, you > > will want > > > > > to use the YARN utilities (yarn application -kill ) to stop > > the YARN > > > > > session. However, the application id in my logs is not found in the > > > > > Resource Manager so it seems to already have been killed (due to the > > > > > original yarn session command not properly executing?). > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/yarn_setup.html#detached-yarn-session > > > > > > > > > Hey Gary, thanks for reaching out. > > > Executing "yarn application -list" does not return my flink cluster so I > > assume like in my initial post that the application must be terminated. My > > config when I ran the job did not have log aggregation enabled and that > > might be why when I try to look at the logs via "yarn logs -applicationId > > ", I get back nothing (e.g. Unable to get ApplicationState. > > Attempting to fetch logs directly from the filesystem. > > > /tmp/logs/si022833/logs/application_1545041832015_73428 does not exist.) > > > > > >
Re: How to shut down Flink Web Dashboard in detached Yarn session?
Hi Sai, could you check that the dashboard you are seeing is really running on Yarn and not a standalone Flink cluster which you have running locally? Cheers, Till On Mon, Dec 31, 2018 at 7:40 PM Sai Inampudi wrote: > Hey Gary, thanks for reaching out. > > Executing "yarn application -list" does not return my flink cluster so I > assume like in my initial post that the application must be terminated. My > config when I ran the job did not have log aggregation enabled and that > might be why when I try to look at the logs via "yarn logs -applicationId > ", I get back nothing: > (e.g. Unable to get ApplicationState. Attempting to fetch logs > directly from the filesystem. > /tmp/logs/si022833/logs/application_1545041832015_73428 does not > exist.) > > > (My previous reply was formatted incorrectly so I am replying back with > proper formatting. Apologies for the mistake) > > > On 2018/12/31 18:13:05, Sai Inampudi wrote: > > > > > > On 2018/12/31 10:53:58, Gary Yao wrote: > > > Hi, > > > > > > You can use the YARN client to list all applications on your YARN > cluster: > > > > > > yarn application -list > > > > > > If this does not show any running applications, the Flink cluster must > have > > > somehow terminated. If you have YARN's log aggregation enabled, you > should > > > be > > > able to view the Flink logs by running: > > > > > > yarn logs -applicationId > > > > > > Best, > > > Gary > > > > > > On Fri, Dec 28, 2018 at 9:42 PM Sai Inampudi > wrote: > > > > > > > Hi everyone, > > > > > > > > I recently attempted to create a Flink cluster on YARN by executing > the > > > > following: > > > > ~/flink-1.5.4/bin/yarn-session.sh -n 5 -tm 2048 -s 4 -d -nm > flink_yarn > > > > > > > > The resulting command was not completely successful but it did end up > > > > creating a Apache Flink Dashboard with 1 Task Manager, 1 Task Slot, > and 1 > > > > Job Manager. > > > > > > > > When I look at my Yarn Resource Manager, I don't see my application > > > > running. CLI calls for the application id also returned nothing. > > > > > > > > I would like to kill the existing web dashboard as well as the other > > > > lingering task manager/job manager so that I can try recreating the > yarn > > > > session successfully. > > > > > > > > Has anyone encountered this before and has any suggestion? I looked > > > > through documentation [1] which says to stop a yarn session, you > will want > > > > to use the YARN utilities (yarn application -kill ) to stop > the YARN > > > > session. However, the application id in my logs is not found in the > > > > Resource Manager so it seems to already have been killed (due to the > > > > original yarn session command not properly executing?). > > > > > > > > > > > > > > > > > > > > [1] > > > > > https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/yarn_setup.html#detached-yarn-session > > > > > > > Hey Gary, thanks for reaching out. > > Executing "yarn application -list" does not return my flink cluster so I > assume like in my initial post that the application must be terminated. My > config when I ran the job did not have log aggregation enabled and that > might be why when I try to look at the logs via "yarn logs -applicationId > ", I get back nothing (e.g. Unable to get ApplicationState. > Attempting to fetch logs directly from the filesystem. > > /tmp/logs/si022833/logs/application_1545041832015_73428 does not exist.) > > >
Re: How to shut down Flink Web Dashboard in detached Yarn session?
Hey Gary, thanks for reaching out. Executing "yarn application -list" does not return my flink cluster so I assume like in my initial post that the application must be terminated. My config when I ran the job did not have log aggregation enabled and that might be why when I try to look at the logs via "yarn logs -applicationId ", I get back nothing: (e.g. Unable to get ApplicationState. Attempting to fetch logs directly from the filesystem. /tmp/logs/si022833/logs/application_1545041832015_73428 does not exist.) (My previous reply was formatted incorrectly so I am replying back with proper formatting. Apologies for the mistake) On 2018/12/31 18:13:05, Sai Inampudi wrote: > > > On 2018/12/31 10:53:58, Gary Yao wrote: > > Hi, > > > > You can use the YARN client to list all applications on your YARN cluster: > > > > yarn application -list > > > > If this does not show any running applications, the Flink cluster must have > > somehow terminated. If you have YARN's log aggregation enabled, you should > > be > > able to view the Flink logs by running: > > > > yarn logs -applicationId > > > > Best, > > Gary > > > > On Fri, Dec 28, 2018 at 9:42 PM Sai Inampudi wrote: > > > > > Hi everyone, > > > > > > I recently attempted to create a Flink cluster on YARN by executing the > > > following: > > > ~/flink-1.5.4/bin/yarn-session.sh -n 5 -tm 2048 -s 4 -d -nm flink_yarn > > > > > > The resulting command was not completely successful but it did end up > > > creating a Apache Flink Dashboard with 1 Task Manager, 1 Task Slot, and 1 > > > Job Manager. > > > > > > When I look at my Yarn Resource Manager, I don't see my application > > > running. CLI calls for the application id also returned nothing. > > > > > > I would like to kill the existing web dashboard as well as the other > > > lingering task manager/job manager so that I can try recreating the yarn > > > session successfully. > > > > > > Has anyone encountered this before and has any suggestion? I looked > > > through documentation [1] which says to stop a yarn session, you will want > > > to use the YARN utilities (yarn application -kill ) to stop the > > > YARN > > > session. However, the application id in my logs is not found in the > > > Resource Manager so it seems to already have been killed (due to the > > > original yarn session command not properly executing?). > > > > > > > > > > > > > > > [1] > > > https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/yarn_setup.html#detached-yarn-session > > > > > Hey Gary, thanks for reaching out. > Executing "yarn application -list" does not return my flink cluster so I > assume like in my initial post that the application must be terminated. My > config when I ran the job did not have log aggregation enabled and that might > be why when I try to look at the logs via "yarn logs -applicationId > ", I get back nothing (e.g. Unable to get ApplicationState. > Attempting to fetch logs directly from the filesystem. > /tmp/logs/si022833/logs/application_1545041832015_73428 does not exist.) >
Re: How to shut down Flink Web Dashboard in detached Yarn session?
On 2018/12/31 10:53:58, Gary Yao wrote: > Hi, > > You can use the YARN client to list all applications on your YARN cluster: > > yarn application -list > > If this does not show any running applications, the Flink cluster must have > somehow terminated. If you have YARN's log aggregation enabled, you should > be > able to view the Flink logs by running: > > yarn logs -applicationId > > Best, > Gary > > On Fri, Dec 28, 2018 at 9:42 PM Sai Inampudi wrote: > > > Hi everyone, > > > > I recently attempted to create a Flink cluster on YARN by executing the > > following: > > ~/flink-1.5.4/bin/yarn-session.sh -n 5 -tm 2048 -s 4 -d -nm flink_yarn > > > > The resulting command was not completely successful but it did end up > > creating a Apache Flink Dashboard with 1 Task Manager, 1 Task Slot, and 1 > > Job Manager. > > > > When I look at my Yarn Resource Manager, I don't see my application > > running. CLI calls for the application id also returned nothing. > > > > I would like to kill the existing web dashboard as well as the other > > lingering task manager/job manager so that I can try recreating the yarn > > session successfully. > > > > Has anyone encountered this before and has any suggestion? I looked > > through documentation [1] which says to stop a yarn session, you will want > > to use the YARN utilities (yarn application -kill ) to stop the YARN > > session. However, the application id in my logs is not found in the > > Resource Manager so it seems to already have been killed (due to the > > original yarn session command not properly executing?). > > > > > > > > > > [1] > > https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/yarn_setup.html#detached-yarn-session > > > Hey Gary, thanks for reaching out. Executing "yarn application -list" does not return my flink cluster so I assume like in my initial post that the application must be terminated. My config when I ran the job did not have log aggregation enabled and that might be why when I try to look at the logs via "yarn logs -applicationId ", I get back nothing (e.g. Unable to get ApplicationState. Attempting to fetch logs directly from the filesystem. /tmp/logs/si022833/logs/application_1545041832015_73428 does not exist.)
Re: How to shut down Flink Web Dashboard in detached Yarn session?
Hi, You can use the YARN client to list all applications on your YARN cluster: yarn application -list If this does not show any running applications, the Flink cluster must have somehow terminated. If you have YARN's log aggregation enabled, you should be able to view the Flink logs by running: yarn logs -applicationId Best, Gary On Fri, Dec 28, 2018 at 9:42 PM Sai Inampudi wrote: > Hi everyone, > > I recently attempted to create a Flink cluster on YARN by executing the > following: > ~/flink-1.5.4/bin/yarn-session.sh -n 5 -tm 2048 -s 4 -d -nm flink_yarn > > The resulting command was not completely successful but it did end up > creating a Apache Flink Dashboard with 1 Task Manager, 1 Task Slot, and 1 > Job Manager. > > When I look at my Yarn Resource Manager, I don't see my application > running. CLI calls for the application id also returned nothing. > > I would like to kill the existing web dashboard as well as the other > lingering task manager/job manager so that I can try recreating the yarn > session successfully. > > Has anyone encountered this before and has any suggestion? I looked > through documentation [1] which says to stop a yarn session, you will want > to use the YARN utilities (yarn application -kill ) to stop the YARN > session. However, the application id in my logs is not found in the > Resource Manager so it seems to already have been killed (due to the > original yarn session command not properly executing?). > > > > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/yarn_setup.html#detached-yarn-session >
How to shut down Flink Web Dashboard in detached Yarn session?
Hi everyone, I recently attempted to create a Flink cluster on YARN by executing the following: ~/flink-1.5.4/bin/yarn-session.sh -n 5 -tm 2048 -s 4 -d -nm flink_yarn The resulting command was not completely successful but it did end up creating a Apache Flink Dashboard with 1 Task Manager, 1 Task Slot, and 1 Job Manager. When I look at my Yarn Resource Manager, I don't see my application running. CLI calls for the application id also returned nothing. I would like to kill the existing web dashboard as well as the other lingering task manager/job manager so that I can try recreating the yarn session successfully. Has anyone encountered this before and has any suggestion? I looked through documentation [1] which says to stop a yarn session, you will want to use the YARN utilities (yarn application -kill ) to stop the YARN session. However, the application id in my logs is not found in the Resource Manager so it seems to already have been killed (due to the original yarn session command not properly executing?). [1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/yarn_setup.html#detached-yarn-session
