usbrandon commented on issue #4748: URL: https://github.com/apache/hop/issues/4748#issuecomment-2578500889
Not sure if I intended to have Hop kill itself after some timeout, but the point was that I may build a bash command line that runs a docker container with a python script inside. If anything goes wrong or hangs in the process that Hop let's it start, then there is no recovery but to kill all of Hop. I wish that I could kill off that process that I had hop start in within the pipeline, or workflow, whichever causes the sidecar execution. I agree on Airflow. We are exploring that. I am interested in how SLA's are declared and to what effect. From what I understand they are only an event marker and Airflow does nothing about it but note that such and such an SLA was violated, but it doesn't seem to kill the job or trigger other logic. I am still exploring so I might come back with a different understanding. Please teach me if you know differently. It is very appealing to have a single pane of glass for scheduling. We have a mixture of things executing like serverless functions, scripts that put elements on an SQS queue that cause processing, and then Hop for ingestion of landed files once the scaled up serverless data gathering tasks are done downloading where Hop can see them. Thanks for the consideration on this request. I hope the extra context helps. On Wed, Jan 8, 2025 at 10:41 AM Matt Casters ***@***.***> wrote: > Hi @bamaer <https://github.com/bamaer>. These are great ideas for sure. > The Apache Airflow "Retry" task can do these things, and many enterprise > schedulers as well. What you usually can't do however, is have the Hop > process itself killed if it takes too long to complete. That is what this > issue tries to solve. > Doing a re-start of pipelines and workflows, even remembering until where > a workflow executed succesfully and restarting from that point, is > something that can be done. I wrote something similar for another similar > tool many years ago. Perhaps it deserves its own issue. > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/hop/issues/4748#issuecomment-2578131638>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAJNF5SOAMNKVI4IDSLEW4L2JVIMVAVCNFSM6AAAAABULCSOCWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZYGEZTCNRTHA> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
