Re: [DISCUSS] Proposal for adding Telemetry via Scarf

2024-03-31 Thread Amogh Desai
+1 looks like a good tool which could be super helpful.

* We should have some transparency into the data that is collected or sent
* We should have an option to optionally opt-out

Thanks & Regards,
Amogh Desai


On Sun, Mar 31, 2024 at 7:53 AM Wei Lee  wrote:

> +1 to this. It would be really useful. As long as we can opt out, I think
> we’re good.
>
> Best,
> Wei
>
> > On Mar 31, 2024, at 12:47 AM, Kaxil Naik  wrote:
> >
> > Grammar Correction:
> >
> > We should assume that those who deploy and upgrade Airflow - actually
> read
> >> and take into account what is written in the release notes - especially
> if
> >> they have security guys breathing their necks, similarly as we have to
> >> assume they follow CVE announcements about security issues fixed. If we
> >> are very straightforward and out-going about the change, inform very
> >> clearly how to opt-out, I don't see a big problem with opt-out.
> >
> >
> > I couldn't agree more; even though we shouldn't collect any data that
> > hamper security (and we should aim to do the same), most security
> concerned
> > folks don't just upgrade, and we can rely on them regarding release notes
> > or announcements and we can make it very clear in our announcements too;
> > and in our installation guides.
> >
> > On Sat, 30 Mar 2024 at 16:47, Kaxil Naik  wrote:
> >
> >> Grammar crrection:
> >>
> >>
> >> On Sat, 30 Mar 2024 at 16:43, Kaxil Naik  wrote:
> >>
> >>> Have this at the end of the email too: but if folks don't read until
> the
> >>> end and quoting Maxime from the use-case blog[1]:
> >>>
> >>> "I think people often ask ‘how do I contribute to open source?’, ‘I've
> >>> got to get into the code’, or ‘ I’ve got to be an engineer.’ Actually,
> the
> >>> very simplest thing that you can do is just say, ‘my organization gets
> real
> >>> value from this piece of software.’ There are a bunch of ways to let
> the
> >>> people know about it – and now Scarf is there. If your organization is
> >>> getting a lot of value from a piece of open source software, make sure
> the
> >>> devs know about it."
> >>>
> >>> What kind of edge cases are you thinking about? I don't think it makes
> >>> sense to have "opt-in" at all. As the goal is to collect data for most
> >>> Airflow installations except for those that don't want to give data,
> then
> >>> "opt-out" is the only way to maximize it. As long as we don't collect
> any
> >>> PII data, this is in-compliance as well.
> >>>
> >>> Imagine someone learning Airflow, if they have to opt-in via a config,
> >>> they wouldn't even know or care about it, hence us losing most of the
> data.
> >>> I understand why some orgs & individuals may want to opt-out.
> >>>
> >>> Scarf Provides tracking pixels (essentially an HTML image tag) that you
> >>> can place in your website or product to track visitors to that URL. If
> >>> there were any concerns about Privacy, ASF wouldn't have approved it
> at all.
> >>>
> >>> A few key details to note about the pixel:
> >>>
> >>>
> >>>   - No PII is tracked… Scarf does not capture/retain IP information…
> >>>   this information is discarded by the platform upon
> processing/aggregating
> >>>   - Scarf pixels respect the Do Not Track (DNT) settings of browsers -
> >>>   these users will not be tracked whatsoever.
> >>>
> >>>
> >>> All the ASF projects I had listed (whether they use Scarf gateway or
> >>> Scarf pixel in product) are using opt-out.
> >>>
> >>> 1. Short opt-in period before opt-out. Test this feature with users who
>  trust and if it works great - make it public. I think it's wise to
> handle
>  edge cases and configure collected data more accurately.
> >>>
> >>>
> >>>
> >>> It would be a pixel in the webserver, should affect nothing at all even
> >>> in an air-gapped environment.
> >>>
>  2. It should not affect anything if access to the internet is
> restricted
>  which is default for many companies.
> >>>
> >>>
> >>>
> >>> 100% agreed on the below:
> >>>
>  I think we have a very good blueprint to follow including at least 5
>  other
>  ASF projects that also passed the review of the privacy@asf. And
> while I
>  understand (and concur) the urge for opt-in by default coming from
>  consumer
>  market (where it makes perfect sense) Airflow is not a consumer
>  software and is used in "corporate environment" which has a little
>  different expectations and broad assumption that the company can make
>  decisions on such telemetry on behalf of the employees using it.
> >>>
> >>>
> >>> Couldn't agree more; even though there shouldn't we collect hamper
> >>> security (and we should aim to do the same), most security concerned
> folks
> >>> don't just
> >>> upgrade, and we can rely on them regarding release notes or
> announcements
> >>> and we can make it very clear in our announcements too; and in our
> >>> installation guides.
> >>>
> >>> We should assume that those who deploy and upgrade Airflow - actually
> read
>  and take

[RESULT][VOTE] AIP-64: Keep TaskInstance try history

2024-03-31 Thread Jed Cunningham
Hello Airflow Community,

The vote for AIP-64 has completed and it has been accepted.

14 "+1" binding votes received:
- Jed Cunningham
- Tzu-ping Chung
- Pierra Jeambrun
- Kaxil Naik
- Jens Scheffler
- Jarek Potiuk
- Vikram Koka
- Amogh Desai
- Phani Kumar
- Niko Oliveira
- Brent Bovenzi
- Utkarsh Sharma
- Josh Fell
- Hussein Awala

6 "+1" non-binding received:
- Arita Basu
- Igor Kholopov
- Wei Lee
- Shubham Mehta
- Rahul Vats
- Ankit Chaurasia

Vote thread:
https://lists.apache.org/thread/cdrxd4tsq982gjmbbl32vp2ygt9dxgpk

Thanks everyone that took the time to vote!

Thanks,
Jed


Re: [VOTE] AIP-62 Getting Lineage from Hook Instrumentation

2024-03-31 Thread Jakub Dardziński
+1 (non-binding). It will enable long-awaited enriched support for
PythonOperator

pt., 29 mar 2024 o 22:23 Jarek Potiuk  napisał(a):

> +1 (binding). I think that's a really important missing piece in lineage
> integration.
>
> On Thu, Mar 28, 2024 at 12:47 PM Maciej Obuchowski  >
> wrote:
>
> > Hello,
> > I would like to start a vote on AIP-62: Getting Lineage from Hook
> > Instrumentation, to be implemented into (presumably) Airflow 2.10.
> >
> > The AIP can be found here:
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-62+Getting+Lineage+from+Hook+Instrumentation
> >
> > Discussion Thread:
> > https://lists.apache.org/thread/5chxcp0zjcx66d3vs4qlrm8kl6l4s3m2
> >
> > The vote will last until 2024-04-03 18:00 UTC and until at least 3
> binding
> > votes have been cast.
> >
> > Please vote accordingly:
> >
> > [ ] + 1 approve
> > [ ] + 0 no opinion
> > [ ] - 1 disapprove with the reason
> >
> > Only votes from PMC members and committers are binding, but other members
> > of the community are encouraged to check the AIP and vote with
> > "(non-binding)".
> >
> > Consider this my binding +1 vote.
> >
> > Thanks,
> > Maciej
> >
>