Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]
alizademhdi closed pull request #38628: Fix inital value for _batch_id attribute in livy operator URL: https://github.com/apache/airflow/pull/38628 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]
github-actions[bot] commented on PR #38628: URL: https://github.com/apache/airflow/pull/38628#issuecomment-2131753717 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]
mateuslatrova commented on PR #38628: URL: https://github.com/apache/airflow/pull/38628#issuecomment-2048507160 Thanks for the idea, @potiuk! That makes much more sense. I will try to implement it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]
potiuk commented on PR #38628: URL: https://github.com/apache/airflow/pull/38628#issuecomment-2036432683 Trigger should (even if it times out) produce a trigger event with batch_id information in it. So Airflow should not kill trigger, it should timeout itself before Airflow does it and signal timeout operation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]
mateuslatrova commented on PR #38628: URL: https://github.com/apache/airflow/pull/38628#issuecomment-2035812370 Hi! I was also working on issue #37898 , and one of the requirements the author mentioned was: > "The Livy operator should fail due to a timeout, and simultaneously, it should kill the Spark job." The way you solved it makes the task fail, but it does not kill the Spark Job. When the task is rewaked up after the trigger times out its `_batch_id` is `None`. So when the task's `kill` method is called, it will not call the `self.hook.delete_batch` method inside the `kill` method. I am trying to figure out a way to get that `_batch_id` when the task is rewaked up so that we could kill the Spark job, but it doesn't seem trivial... Any ideias? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]
alizademhdi commented on PR #38628: URL: https://github.com/apache/airflow/pull/38628#issuecomment-2030790977 Sure. I will try. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]
eladkal commented on PR #38628: URL: https://github.com/apache/airflow/pull/38628#issuecomment-2028687443 Can you add unit test to avoid regression? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]
alizademhdi commented on PR #38628: URL: https://github.com/apache/airflow/pull/38628#issuecomment-2028073596 Fixed static tests :D -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Fix inital value for _batch_id attribute in livy operator [airflow]
alizademhdi opened a new pull request, #38628: URL: https://github.com/apache/airflow/pull/38628 ### Related issue #37898 ### Explain when the livy operator is rewaked up after trigger is time out, the _batch_id is not initialized, which cause it to raise this exeption: `AttributeError: 'LivyOperator' object has no attribute '_batch_id'` this problem arises because an initial value is not defined for _batch_id. by initializing this attribute to None, this problem will be solved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
