Hello, I am a newbie at Yunikorn and I was doing some test deployments to investigate how integrate Spark (without Operator) with Yunikorn in Kubernetes. I was able to link both applications and also to see Spark Jobs/Apps from Yunikorn Web UI including new labeling and tunneling capacities by K8S config but there is a problem with the Status App showed in Yunikorn Web UI. I can only see 3 states: completed, running or new, despite some Spark Jobs fails. I was wondering why I can not see real state of Spark App, if it fails (I can see it failed on K8S environment) it showed with state completed. It is supposed that the connection between k8s and Yunikorn might provide that information, am I right? Or I am not considering some important aspect at the time of the deployment of my spark jobs or Yunikorn?
Some information about my test deployment: * Spark v3.1.2 apps (via k8s jobs or cronjobs) * Yunikorn v1.1.0 (via Helm) * Openshift k8s environment * Including these properties in the spark-submit apps: * --conf spark.kubernetes.driver.scheduler.name=yunikorn * --conf spark.kubernetes.executor.scheduler.name=yunikorn * --conf spark.kubernetes.executor.label.applicationId=spark-app-01 * --conf spark.kubernetes.driver.label.applicationId=spark-app-01 * --conf spark.kubernetes.executor.label.queue=root.test-dev * --conf spark.kubernetes.driver.label.queue=root.test-dev I hope all the information of my case study is helpful and I would be delighted to give more details if were needed. Also sorry to send this e-mail directly to all of you but I have not found any documentation or resource to avoid this problem. Many thanks in advance and best regards, Daniel ________________________________ Este correo electr?nico y, en su caso, cualquier fichero anexo al mismo, contiene informaci?n de car?cter confidencial exclusivamente dirigida a su destinatario o destinatarios. Si no es vd. el destinatario indicado, queda notificado que la lectura, utilizaci?n, divulgaci?n y/o copia sin autorizaci?n est? prohibida en virtud de la legislaci?n vigente. En el caso de haber recibido este correo electr?nico por error, se ruega notificar inmediatamente esta circunstancia mediante reenv?o a la direcci?n electr?nica del remitente. Evite imprimir este mensaje si no es estrictamente necesario. This email and any file attached to it (when applicable) contain(s) confidential information that is exclusively addressed to its recipient(s). If you are not the indicated recipient, you are informed that reading, using, disseminating and/or copying it without authorisation is forbidden in accordance with the legislation in effect. If you have received this email by mistake, please immediately notify the sender of the situation by resending it to their email address. Avoid printing this message if it is not absolutely necessary.
