Hi Sung, Thanks for the clarification, with that information in mind I think I can more accurately summarize my concern with this AIP as it is currently presented.
Deprecating one feature "Task SLAs" and introducing a different feature "DAG SLAs" seems like two fairly independent choices, not one. From my experience of using Airflow in different contexts and companies DAG SLA uses cases would not overlap enough with Task SLA use cases (which have been compelling enough for some users such as myself to put up with the existing brokenness/misfeatureness) and therefore if Task SLAs are deprecated it does not help that DAG SLAs have been introduced. I appreciate that Task SLAs are taxing for Airflow devs and deprecating them helps maintainability and support bandwidth, in my opinion this should be a strong enough reason on it's own to take this action. Damian -----Original Message----- From: Sung Yun <[email protected]> Sent: Monday, March 20, 2023 2:23 PM To: [email protected] Subject: Re: [DISCUSS] Airflow - New SLA AIP Hi Damian, It’s great to have your feedback! And I think it’s especially great that we are gaining feedback from people who are using the current sla feature. To answer your points: 1. I never made that assumption at all. I think the reason Ash suggested that we put it up for votes and the reason we are marking this proposal for a Major version update, is because we are aware that there may be current users making use of the existing feature, no matter how broken it is. 2. That is absolutely correct. And that is the reason why I’m giving an example of how people will be able to write up their own sla checking tasks that will be more responsive than the existing misfeature. My proposal only highlights all the problems that are in place, why I think that these problems are going to be impossible to fix when we support task-level slas, and why I am proposing a DAG-level SLA, given how much more correct, lightweight and simpler it will be to implement in Airflow. I personally don’t have a strong opinion on whether we should be deleting the existing misfeature, mostly out of concern from the pushback we might be getting from any existing users. However, I can definitely empathize in the dev community in wanting to deprecate the existing misfeature, because keeping a misfeature is costly to the community. It’s costly to new users who are wasting time figuring out why it’s not behaving as they expect. It’s costly to contributors trying to figure out how they could fix it. And last but not least, it’s costly to the maintainers who need to repeatedly respond to questions asking why the feature is broken. Sent from my iPhone > On Mar 20, 2023, at 1:42 PM, Damian Shaw <[email protected]> wrote: > > Thanks, I've caught up on the discussion and the word document, while I > agree with a lot of the reasoning I have two pieces of feedback: > > 1. The document seems to assume that the current SLAs are sufficiently broken > that no one is depending on them, this isn't the case, I am at least 1 user > depending on the current set-up and have put efforts in to making sure there > is better documentation around it's behavior: > https://github.com/apache/airflow/pull/27111 > > 2. The most important use case for my set-up is that different tasks in the > same DAG can have different SLAs, e.g. I need an SLA if 1 task does not > complete 1 hour after data_interval_end, and a different SLA of 3 hours for a > different task in the same DAG. As a DAG may contain many 1000s of tasks but > only a small subset of be critical enough to warrant an SLA, I imagine this > would be a widespread use case. Am I understanding it correctly that this use > case will no longer be supported? Please correct me if I am misreading this > document. > > Thanks, > Damian > > > > -----Original Message----- > From: Jarek Potiuk <[email protected]> > Sent: Sunday, March 19, 2023 5:22 PM > To: [email protected] > Subject: Re: [DISCUSS] Airflow - New SLA AIP > > You can see all the discussions in > https://lists.apache.org/[email protected]. The SLA discussion > is in progress in the google docs for now (soon to be converted into cwiki > AIP). > > J, > >> On Sun, Mar 19, 2023 at 10:15 PM Damian Shaw >> <[email protected]> wrote: >> >> I'm missing the previous conversation / who this is replying to and I don't >> see a new AIP yet, is this still in progress? >> >> I would definitely want to be able provide feedback on an AIP for SLAs, >> while the current SLA system is limited I am making heavy usage of it. >> >> -----Original Message----- >> From: Jarek Potiuk <[email protected]> >> Sent: Sunday, March 19, 2023 4:29 PM >> To: [email protected] >> Subject: Re: [DISCUSS] Airflow - New SLA AIP >> >>> Jarek - thank you for the suggestions - those are great suggestions in >>> enhancing the proposal... Regarding your first point - how does deprecating >>> an existing feature work for Airflow? Do we need to give a grace period of >>> a minor or major patch over which we continue to support the feature after >>> marking it for deprecation? Or do we remove them right away on a major >>> version / minor version update? >> >> We raise a deprecation warning. You can see a number of examples in a number >> of places. We cannot remove them in minor. We have not yet discussed the >> timeline for Airflow 3 - but that is the first time when things can get >> removed. >> >>> Also, do you folks have any thoughts on my last point listed in 'Other >>> Considerations' section? I've highlighted the section to make it easier to >>> identify, and I think it's an edge case that would be worth thinking about. >> >> responded in the doc. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> ________________________________ >> Strike Technologies, LLC (“Strikeâ€) is part of the GTS family of >> companies. Strike is a technology solutions provider, and is not a broker or >> dealer and does not transact any securities related business directly >> whatsoever. This communication is the property of Strike and its affiliates, >> and does not constitute an offer to sell or the solicitation of an offer to >> buy any security in any jurisdiction. It is intended only for the person to >> whom it is addressed and may contain information that is privileged, >> confidential, or otherwise protected from disclosure. Distribution or >> copying of this communication, or the information contained herein, by >> anyone other than the intended recipient is prohibited. If you have received >> this communication in error, please immediately notify Strike at >> [email protected], and delete and destroy any copies hereof. >> ________________________________ >> >> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments >> are intended solely for the addressee. This transmission is covered by the >> Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The information >> contained in this transmission is confidential in nature and protected from >> further use or disclosure under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338 >> (1999), and may be subject to attorney-client or other legal privilege. Your >> use or disclosure of this information for any purpose other than that >> intended by its transmittal is strictly prohibited, and may subject you to >> fines and/or penalties under federal and state law. If you are not the >> intended recipient of this transmission, please DESTROY ALL COPIES RECEIVED >> and confirm destruction to the sender via return transmittal. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > ________________________________ > Strike Technologies, LLC (“Strikeâ€) is part of the GTS family of > companies. Strike is a technology solutions provider, and is not a broker or > dealer and does not transact any securities related business directly > whatsoever. This communication is the property of Strike and its affiliates, > and does not constitute an offer to sell or the solicitation of an offer to > buy any security in any jurisdiction. It is intended only for the person to > whom it is addressed and may contain information that is privileged, > confidential, or otherwise protected from disclosure. Distribution or copying > of this communication, or the information contained herein, by anyone other > than the intended recipient is prohibited. If you have received this > communication in error, please immediately notify Strike at > [email protected], and delete and destroy any copies hereof. > ________________________________ > > CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments are > intended solely for the addressee. This transmission is covered by the > Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The information > contained in this transmission is confidential in nature and protected from > further use or disclosure under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338 > (1999), and may be subject to attorney-client or other legal privilege. Your > use or disclosure of this information for any purpose other than that > intended by its transmittal is strictly prohibited, and may subject you to > fines and/or penalties under federal and state law. If you are not the > intended recipient of this transmission, please DESTROY ALL COPIES RECEIVED > and confirm destruction to the sender via return transmittal. > B‹KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKCB•È [œÝXœØÜšX™KK[XZ[ ˆ]‹][œÝXœØÜšX™PZ\™›Ý˘\XÚK›Ü™ÃB‘›ÜˆY][Û˜[ÛÛ[X[™ ËK[XZ[ ˆ]‹Z[ Z\™›Ý˘\XÚK›Ü™ÃB --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected] ________________________________ Strike Technologies, LLC (“Strike”) is part of the GTS family of companies. Strike is a technology solutions provider, and is not a broker or dealer and does not transact any securities related business directly whatsoever. This communication is the property of Strike and its affiliates, and does not constitute an offer to sell or the solicitation of an offer to buy any security in any jurisdiction. It is intended only for the person to whom it is addressed and may contain information that is privileged, confidential, or otherwise protected from disclosure. Distribution or copying of this communication, or the information contained herein, by anyone other than the intended recipient is prohibited. If you have received this communication in error, please immediately notify Strike at [email protected], and delete and destroy any copies hereof. ________________________________ CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments are intended solely for the addressee. This transmission is covered by the Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The information contained in this transmission is confidential in nature and protected from further use or disclosure under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338 (1999), and may be subject to attorney-client or other legal privilege. Your use or disclosure of this information for any purpose other than that intended by its transmittal is strictly prohibited, and may subject you to fines and/or penalties under federal and state law. If you are not the intended recipient of this transmission, please DESTROY ALL COPIES RECEIVED and confirm destruction to the sender via return transmittal. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
