Re: Request for feedback on proposal for new OpenLineage provider in Airflow

2023-01-27 Thread Jarek Potiuk
General comment from my side: I think Open Lineage is (and should be even more) a feature of Airflow that expands Airflow's capabilities greatly and opens up the direction we've been all working on - Airflow as a Platform. I think closely integrating it with Open-Lineage goes the same direction

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

2023-01-26 Thread Jarek Potiuk
Yeah. The PR was mostly to bring things together- i.e to be able to say "this is really a public interface of ours". And by just doing it, I was actually hoping it will spark those kind of discussions where we will really detail on what it means for each of those to be public. Having it written

Re: [VOTE] Airflow Providers prepared on January 23, 2023

2023-01-23 Thread Jarek Potiuk
Agree. On Mon, Jan 23, 2023 at 8:18 PM Elad Kalif wrote: > > I already had cassandra ready in the branch before the fix so I just > continued with it. > regardless we would have released it in the upcoming full wave so it doesn't > really matter. > > On Mon, Jan 23,

Re: [VOTE] Airflow Providers prepared on January 23, 2023

2023-01-23 Thread Jarek Potiuk
+1 (binding) - tested my changes, signatures, checksums, licences all looks good. One note - we do not technically have to release cassandra as it is virtually the same as the previous package. While it contains two changes of mine (one adding dnspython limitation and second removing it after the

Re: [VOTE] Release Airflow Python Client 2.5.0 based on 2.5.0rc1

2023-01-20 Thread Jarek Potiuk
+1 (binding) - tested signatures/checksums/licences. On Fri, Jan 20, 2023 at 6:46 PM Pierre Jeambrun wrote: > +1 (non-binding) > > Le lun. 16 janv. 2023 à 13:23, Ephraim Anierobi < > ephraimanier...@apache.org> a écrit : > >> Hello everyone, >> >> I have cut the first release candidate for the

Re: [VOTE][ISSUE-22073] Finalising approach for displaying non-ascii characters in DAG display name

2023-01-19 Thread Jarek Potiuk
e do not even know). I would rather attempt it in a pragmatic way - even if it means we need to add an extra "display" entity. Seems like more predictable in terms of problems our users will experience. J, On Mon, Jan 16, 2023 at 9:19 PM Jarek Potiuk wrote: > > Possibly contentious

Re: [VOTE] Release Airflow 2.5.1 from 2.5.1rc2

2023-01-19 Thread Jarek Potiuk
We need a third PMC vote :D On Thu, Jan 19, 2023 at 10:53 PM Pankaj Singh wrote: > +1 (non-binding) > > Run units, integrations and example DAG's of our astro-sdk-python with > 2.5.1rc2 package. > > On Thu, Jan 19, 2023 at 9:41 PM Jarek Potiuk wrote: > >> +

Re: [VOTE] Release Airflow 2.5.1 from 2.5.1rc2

2023-01-19 Thread Jarek Potiuk
+1 (binding) tested few dags, signature, checksums, licences - looks good. On Wed, Jan 18, 2023 at 11:01 PM Pierre Jeambrun wrote: > Hey fellow Airflowers, > > I have cut Airflow 2.5.1rc2. This email is calling a vote on the release, > which will last *24 hours*, from Wednesday, January 18,

Re: [VOTE][ISSUE-22073] Finalising approach for displaying non-ascii characters in DAG display name

2023-01-16 Thread Jarek Potiuk
or Mysql we enforce it as ASCII only. > > On Jan 12 2023, at 6:15 pm, Jarek Potiuk wrote: > > As I mentioned multiple times in similar discussions We have a huge > problem with unicode in dag_id. Namely MySQL limit on indexes. We would > have to shorten the Id significantly in the

Re: [VOTE] Airflow Providers prepared on January 14, 2023

2023-01-16 Thread Jarek Potiuk
:D On Mon, Jan 16, 2023 at 9:06 PM Oliveira, Niko wrote: > > One comment Niko - for software release voting only PMC votes are > binding :) > > > Oops, sorry, too eager XD > -- > *From:* Jarek Potiuk > *Sent:* Monday, January 16, 2

Re: [VOTE] Airflow Providers prepared on January 14, 2023

2023-01-16 Thread Jarek Potiuk
One comment Niko - for software release voting only PMC votes are binding :) +1 (binding) - tested signatures, checksums, licences. all looks good. On Mon, Jan 16, 2023 at 7:42 PM Oliveira, Niko wrote: > +1 binding > > Our system tests show the AWS provider package as stable, a couple known >

Re: [VOTE] Release Airflow 2.5.1 from 2.5.1rc1

2023-01-16 Thread Jarek Potiuk
+1 (binding) . Tested (almost) all changes I've been involved in, checked sources, signatures, licences, run a few dags. Looks cool On Sat, Jan 14, 2023 at 10:46 PM Pierre jeambrun wrote: > Hey fellow Airflowers, > > We have cut Airflow 2.5.1rc1. This email is calling a vote on the release, >

Re: [VOTE][ISSUE-22073] Finalising approach for displaying non-ascii characters in DAG display name

2023-01-12 Thread Jarek Potiuk
As I mentioned multiple times in similar discussions We have a huge problem with unicode in dag_id. Namely MySQL limit on indexes. We would have to shorten the Id significantly in the database to workaround MySQL limits for index size. We can have a wishful thinking that we can change dag_id to

Re: [VOTE] AIP-52 Automatic setup and teardown tasks

2023-01-09 Thread Jarek Potiuk
+1 (binding) On Mon, Jan 9, 2023 at 6:01 PM Ferruzzi, Dennis wrote: > +1 non-binding > > -- > *From:* Ash Berlin-Taylor > *Sent:* Monday, January 9, 2023 8:27 AM > *To:* dev@airflow.apache.org > *Subject:* [EXTERNAL] [VOTE] AIP-52 Automatic setup and teardown tasks

Re: [VOTE] Airflow Providers prepared on January 02, 2023 - RC2

2023-01-04 Thread Jarek Potiuk
+1 (binding) - check signatures, checksums, licenses and all verified that all the issues I've fixed are fixed in the binary wheel packages. On Wed, Jan 4, 2023 at 8:15 AM Phani Kumar wrote: > +1 (non-binding) > > On Wed, Jan 4, 2023 at 2:56 AM Jed Cunningham > wrote: > >> +1 (binding) >> >>

Re: [PROPOSAL] Switching our CI runners to K8S controller open-sourced for Apache Arrow

2022-12-21 Thread Jarek Potiuk
Cool! On Wed, Dec 21, 2022 at 5:57 PM Jed Cunningham wrote: > I'll help. I've been wanting to get more familiar with our runners, so > helping with their successor is the perfect opportunity. > > I don't have a strongly held opinion on the move to the new controller, > but it seems reasonable

Re: New provider for SAS

2022-12-20 Thread Jarek Potiuk
e there any other providers which have done this so that we > could model it in a similar manner) > > > > Thank you for your time. > > > > *From: *Jarek Potiuk > *Date: *Tuesday, December 20, 2022 at 3:38 PM > *To: *dev@airflow.apache.org > *Subject: *Re: New pr

Re: New provider for SAS

2022-12-20 Thread Jarek Potiuk
ic documented SAS Viya APIs available through the SAS Developer > Portal: https://developer.sas.com/apis/rest/ > > > > Please let me know if there are additional concerns. I will submit a PR as > well so you can take a look at what we have. > > > > Thanks > > &g

Re: [PROPOSAL] (slightly) change approach to default config values

2022-12-20 Thread Jarek Potiuk
> Are you planning to enforce this pattern when new parameters are introduced? It will be enforced semi-automatically. 1) If you are an existing user, new parameters will work out-of-the-box. Default values will be read from internal values, those new parameters will be either missing in the old

[PROPOSAL] (slightly) change approach to default config values

2022-12-20 Thread Jarek Potiuk
Hello everyone, I have a proposal (Draft PR here: https://github.com/apache/airflow/pull/28495) how we can change the approach to default values for Airflow configuration - namely to keep "defaults" internally in airflow application rather than with generated "airflow.cfg" file. This has been

Re: [Discuss]whether fail early for Xcom.set when value exceeds the BLOB limit

2022-12-20 Thread Jarek Potiuk
+1. I am all for failing it. I do not see an immediate issue with it - especially if we also explain how to handle it (i.e. custom XCom, limiting data, truncating data). I assume this is only when "standard" XCom is used. I would be for it, explicit is better than implicit. The XCom limit has

Re: [ANNOUNCE] New committer Niko Oliveira (o-nikolas)

2022-12-19 Thread Jarek Potiuk
Congrats Niko! On Mon, Dec 19, 2022 at 10:22 PM Elad Kalif wrote: > Hello Airflow Community, > > I am happy to announce that the Project Management Committee (PMC) for > Apache Airflow > has invited Niko Oliveira (github nickname o-nikolas) to become a > committer and we are pleased to announce

Re: [DISCUSSION] New Provider: Pandera

2022-12-18 Thread Jarek Potiuk
Sorry to keep you waiting that long. I am catching up with some old / unhandled threads. > One major con is it won't be part of Airflow constraints file. What is the implication of this? If you have many dependencies that can conflict with other providers, we are publishing "constraints" that

[PROPOSAL] Switching our CI runners to K8S controller open-sourced for Apache Arrow

2022-12-18 Thread Jarek Potiuk
Hello everyone, TL;DR: I wanted to make a proposal to move our CI runners from our own "custom" implementation developed mostly by Ash and based on VMs to a newly released Auto-scaling K8S controller that was developed for Apache Arrow by Voltron Data. I was in contact with Jacob Wujciak who

Re: [DISCUSS] AIP-52 Automatic setup and tear down tasks

2022-12-16 Thread Jarek Potiuk
Love it. I think it solves quite a number of existing cases but - more importantly - opens up the possibility of even more cases. The Setup and Teardown tasks might be also used to manage the lifecycle of Dags and Tasks by DAG authors for a number of other cases. I think one such case (taken

Re: New provider for SAS

2022-12-16 Thread Jarek Potiuk
Yep. I think the important one is that I believe (or at least this is my proposal to address some of the concerns of the maintainers) we will be leaning towards the approach that if we accept any submission for an "external service" provider, then there must be a commitment from the service

Re: [LAZY CONSENSUS] Release of API Clients

2022-12-14 Thread Jarek Potiuk
Good point. We should document such things every time in READMEs - and in this case it should not only be READMEs but also appropriate steps in the release process IMHO. https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md Initially it could be pretty vague - we usually

Re: [PROPOSAL] (Internal) move of provider packages to isolated "providers" sub-folders

2022-12-14 Thread Jarek Potiuk
s well. I was basically > suggesting move the system tests for a given provider into that provider's > tests module, alongside hooks, operators, sensors, etc: > > tests/providers/airbyte/hooks > tests/providers/airbyte/operators > tests/providers/airbyte/sensors > tests/providers/airbyte/syste

Re: [VOTE] Airflow Providers prepared on December 14, 2022 (RC1)

2022-12-14 Thread Jarek Potiuk
" gpg: Good signature from "Elad Kalif " [unknown] On Wed, Dec 14, 2022 at 10:15 AM Jarek Potiuk wrote: > +1 (binding) - tested licences, signatures, checksums. Verified that the > changes I implemented are in the release. > > On Wed, Dec 14, 2022 at 1:01 AM

Re: [VOTE] Airflow Providers prepared on December 14, 2022 (RC1)

2022-12-14 Thread Jarek Potiuk
+1 (binding) - tested licences, signatures, checksums. Verified that the changes I implemented are in the release. On Wed, Dec 14, 2022 at 1:01 AM Elad Kalif wrote: > Hey all, > > > > I have just cut ad-hoc wave Airflow Providers packages. This email is > calling a vote on the release, > >

Re: [VOTE] New Provider: Cloudera

2022-12-13 Thread Jarek Potiuk
is very easy - just click "Suggest a change on this page" and you will get PR opened, where you will be able to link to your provider. J. On Tue, Dec 13, 2022 at 4:21 PM Jarek Potiuk wrote: > The documentation index is only for the community and only for providers > managed

Re: [VOTE] New Provider: Cloudera

2022-12-13 Thread Jarek Potiuk
; >>>> Ash point is echoing in me, remembering when I had to work on a >>>> specific provider where free accounts/quotas were not available. It was >>>> basically a shot in the dark, making code changes based on documentation >>>> and api specs without be

Re: [PROPOSAL] (Internal) move of provider packages to isolated "providers" sub-folders

2022-12-12 Thread Jarek Potiuk
byte provider is moved | src/airflow/providers/airbyte/hooks/airbyte.py <- this is actual path "inside" the provider So there is far less redundancy than you think. On Tue, Dec 13, 2022 at 2:40 AM Jarek Potiuk wrote: > >> I have two imm

Re: [PROPOSAL] (Internal) move of provider packages to isolated "providers" sub-folders

2022-12-12 Thread Jarek Potiuk
> > > I have two immediate thoughts on it. First, can this be used as an > opportunity to reorganize the system tests tree into the provider's test > tree? Instead of having `tests/system/providers/{foo}`, maybe we can move > those system tests alongside the other provider tests like this: >

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

2022-12-12 Thread Jarek Potiuk
nitely make those discussions > easier. > > > ------ > *From:* Jarek Potiuk > *Sent:* Thursday, December 8, 2022 10:57 AM > *To:* dev@airflow.apache.org > *Subject:* RE: [EXTERNAL][DISCUSSION] Assessing what is a breaking change > for Airflow (Se

[PROPOSAL] (Internal) move of provider packages to isolated "providers" sub-folders

2022-12-11 Thread Jarek Potiuk
Hello everyone, TL;DR: I have a proposal to reorganise the way how our providers are kept in our sources to reflect more the standard ways of packaging of the providers. It's a REALLY long one, so If you are not ready to deep dive in discussion on the structure of the airflow project, you might

Re: [VOTE] New Provider: Cloudera

2022-12-09 Thread Jarek Potiuk
ed up. > > Will the community really be able to contribute and support the provider, > while most of us don't have a paid account ? Or is it 'stakeholders' > maintained and 'community' released at most. (Even reviewing code for > release would be tricky without an account). > &g

Re: [VOTE] New Provider: Cloudera

2022-12-09 Thread Jarek Potiuk
t; So it's not a veto (as vetos can only apply to code). > > My concern about how we will actually test it works given we'd need a > cloudera account/install/instance would be good to comment on though. > > -ash > > On Dec 7 2022, at 1:43 pm, Jarek Potiuk wrote: > > Yeah

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

2022-12-08 Thread Jarek Potiuk
en codifying it in an AIP. > At the moment we consider even the deepest/smallest "private" helper > function within util provider code to be public. This level of public API > makes iterating and maintaining the code very laborious. So I definitely > think this is worth th

Re: [PROPOSAL] Dealing with public runner test failues (Integration tests restructuring)

2022-12-08 Thread Jarek Potiuk
t;> improvements, let me know if I can help. >> >> Cheers, >> Niko >> -- >> *From:* Jarek Potiuk >> *Sent:* Tuesday, December 6, 2022 5:54:07 AM >> *To:* dev@airflow.apache.org >> *Subject:* [EXTERNAL] [PROPOSAL] Dealing w

Re: [PROPOSAL] Dealing with public runner test failues (Integration tests restructuring)

2022-12-07 Thread Jarek Potiuk
should be a bit faster). J. On Wed, Dec 7, 2022 at 6:36 PM Oliveira, Niko wrote: > Awesome to hear this! > > I was really battling this issue last week, very excited for these > improvements, let me know if I can help. > > Cheers, > Niko > -----

Re: [VOTE] New Provider: Cloudera

2022-12-07 Thread Jarek Potiuk
letting ourselves in for inthe long run with an ever growing number of >> providers. Particularly one that needs paid-for accounts that we don't have >> access to! >> >> -ash >> >> On Dec 4 2022, at 11:59 pm, Kaxil Naik wrote: >> >> +1 binding >> &g

[PROPOSAL] Dealing with public runner test failues (Integration tests restructuring)

2022-12-06 Thread Jarek Potiuk
Hey everyone, I think many contributors (non-committers) started to suffer from often failing (disappearing) test runs (mostly for sqlite). Together with @Taragolis, we looked at those recent stability issues with "public runners". They all boil down to the integration tests taking too much

Re: [DISCUSSION] New Provider: Pandera

2022-12-05 Thread Jarek Potiuk
With tools like Pandera, which are (as I understand) pretty standalone and easy to fully automatically test (I believe it can be fully testable with Pandas dataframes), In general I think personally (but that's my personal opinion) we should have no problem with approving a new provider as

Re: Proposal to Remove Executor Coupling in Core Airlfow Code Base

2022-12-05 Thread Jarek Potiuk
Cool! On Mon, Dec 5, 2022 at 9:16 PM Oliveira, Niko wrote: > Hey folks! > > As a follow-up, if you're interested in following along with this project > or even taking some tasks, I've created a dashboard using Github's new > Projects tool. You can see the backlog of tasks, who's assigned, the >

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

2022-12-05 Thread Jarek Potiuk
have references. > > Again I just skimmed it so I am shooting from the hip as they say ;-) > > > > Op za 3 dec. 2022 om 10:25 schreef Jarek Potiuk : >> >> Sorry for not following up on this for a bit - it's been hectic these >> days for me. I think valid points were sa

Re: [DISCUSSION] Release of API clients

2022-12-04 Thread Jarek Potiuk
t; > Best regards, > Pierre > > Le mer. 16 nov. 2022 à 06:17, Sumit Maheshwari a > écrit : >>> >>> Not with every release. I am talking about releasing it with every release >>> "if it is needed". >> >> >> That clarifies, plz

Re: [VOTE] New Provider: Cloudera

2022-12-03 Thread Jarek Potiuk
I think cloudera is important player in our ecosystem and as long as it passes all the bars (i.e. 2.3.0+ compatibility and good non-conflicting dependencies, passing all the tests, I am +1. On Sat, Dec 3, 2022 at 12:51 PM Philippe Lanoe wrote: > > Hello, > > Correction: since it is a vote on

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

2022-12-03 Thread Jarek Potiuk
Sorry for not following up on this for a bit - it's been hectic these days for me. I think valid points were said, and from the tone of those I feel that we all who participated have the same sense of what is important: 1) users "peace of mind" as top priority: clarity of what they can expect

[ANNOUNCE] Airflow Providers released on December 02, 2022 released

2022-12-02 Thread Jarek Potiuk
Dear Airflow community, I'm happy to announce that new versions of Airflow Providers packages were just released. This was an ad-hoc release of three providers with bug-fixes necessary to release Airflow 2.5.0 (also released today): exasol (4.1.2), snowflake(4.0.2) and zendesk (4.2.0).

[RESULT][VOTE] Airflow Providers - release of December 1st, 2022 (RC1)

2022-12-02 Thread Jarek Potiuk
Hello, Apache Airflow Providers (based on RC1) have been accepted. 5 "+1" binding votes received: - Jarek Potiuk (binding) - Jed Cunningham (binding) - Ephraim Anierobi (binding) - Kaxil Naik (binding) - Ash Berlin-Taylor (binding) 3 "+1+ non-binding votes received: - Pank

[ANNOUNCE] New committer Andrey Anshin (Taragolis)

2022-12-01 Thread Jarek Potiuk
Hello Airflow Community, I am happy to announce that the Project Management Committee (PMC) for Apache Airflow has invited Andrey Anshin (github nickname Taragolis) to become a committer and we are pleased to announce that they have accepted. Congratulations Andrey, and welcome! Jarek on behalf

Re: [VOTE] Release Airflow 2.5.0 from 2.5.0rc3

2022-12-01 Thread Jarek Potiuk
+1 binding. Checked signatures, checksums, licences, and ran it via `breeze start airflow` with a number of example DAGs. On Thu, Dec 1, 2022 at 9:02 PM Kaxil Naik wrote: > > +1 (binding) > > > > On Thu, 1 Dec 2022 at 19:02, Jed Cunningham wrote: >> >> +1 (binding) >> >> Checked signatures,

[VOTE] Airflow Providers prepared on December 1st, 2022 (RC1)

2022-12-01 Thread Jarek Potiuk
Hey all, I have just cut the new wave Airflow Providers packages. This email is calling a vote on the release, which will last for 24 hours - which means that it will end on Thursday, 2nd of Dec 2022, 4.30 PM CET. This release is accelerated - because those providers are needed for 2.5.0 release

Re: Status of testing Providers that were prepared on December 01, 2022

2022-12-01 Thread Jarek Potiuk
Wrong subject :( . Sending the right one... On Thu, Dec 1, 2022 at 4:01 PM Jarek Potiuk wrote: > > Hey all, > > I have just cut the new wave Airflow Providers packages. This email is > calling a vote on the release, > which will last for 24 hours - which means that it will end

Status of testing Providers that were prepared on December 01, 2022

2022-12-01 Thread Jarek Potiuk
Hey all, I have just cut the new wave Airflow Providers packages. This email is calling a vote on the release, which will last for 24 hours - which means that it will end on Thursday, 2nd of Dec 2022, 4 PM CET. This release is accelerated - because those providers are needed for 2.5.0 release

Re: [DISCUSS] Allow shorter voting periods for subsequent RCs

2022-11-30 Thread Jarek Potiuk
Good point. The SHOULD is there and after discussions with the ASF folks on JIRAs and members@a.o list - if there is a good reason, we can shorten it. Critical security fix is definitely a good reason for it (happened in Log4J for example) but the community (us) might decide on releasing it

Airflow Providers released on November 29, 2022 are ready

2022-11-29 Thread Jarek Potiuk
Dear Airflow community, I'm happy to announce that new versions of Airflow Providers packages were just released. This is a follow-up release after the November release. Mostly it's about fixing problems we found after release with common.sql provider and its interaction with

[RESULT][VOTE] Airflow Providers prepared on November 26, 2022 (RC3)

2022-11-29 Thread Jarek Potiuk
Hello, Apache Airflow Providers (based on RC3) have been accepted. 3 "+1" binding votes received: - Jarek Potiuk (binding) - Ash Berlin-Taylor (binding) - Ephraim Anierobi (binding) 2 "+1+ non-binding votes received: - Phani Kumar - Pankaj Koti Vote thread: https://lists.ap

Re: [VOTE] Airflow Providers prepared on November 26, 2022 (RC3)

2022-11-29 Thread Jarek Potiuk
cessfully with the below providers. > > apache-airflow-providers-amazon==6.2.0rc3, > apache-airflow-providers-common-sql==1.3.1rc3, > apache-airflow-providers-databricks==4.0.0rc3, > apache-airflow-providers-google==8.6.0rc3, > apache-airflow-providers-snowflake==4.0.1rc3 > >

Re: [VOTE] Release Airflow 2.5.0 from 2.5.0rc2

2022-11-29 Thread Jarek Potiuk
+1 (binding) . Verified licences, signatures, checksums. I tested a few DAGs, looked around the new small UI improvements. I particularly like the draggable grid view split and how log view auto-adjusts (and that it stays sticky between reloads) - this is a huge UX improvement for the Grid view.

Re: [VOTE] Airflow Providers prepared on November 26, 2022 (RC3)

2022-11-28 Thread Jarek Potiuk
for the fix. On Sat, Nov 26, 2022 at 12:06 PM Jarek Potiuk wrote: > > Hey all, > > I have just cut the new wave Airflow Providers packages. This email is > calling a vote on the release, which will last for 72 hours - > which means that it will end on Tue 29 Nov 12:00:00 CET 2022. > Mo

Re: Make KubernetesExecutor's multi_namespace_mode more flexible & enterprise-ready

2022-11-27 Thread Jarek Potiuk
Looks solid. I don't even think it needs more discussion/consensus/voting - that seems like a useful option to have in KubernetesExecutor and just discussing details in PR / committer approval would be enough. On Mon, Nov 21, 2022 at 8:20 PM Ferruzzi, Dennis wrote: > On the surface this sounds

[VOTE] Airflow Providers prepared on November 26, 2022 (RC3)

2022-11-26 Thread Jarek Potiuk
Hey all, I have just cut the new wave Airflow Providers packages. This email is calling a vote on the release, which will last for 72 hours - which means that it will end on Tue 29 Nov 12:00:00 CET 2022. Most of the changes has already been tested and we are past Thanksgiving so it should be

Re: [VOTE] Airflow Providers prepared on November 24, 2022 (RC2)

2022-11-25 Thread Jarek Potiuk
who helped with testing! ) so hopefully the last round of testing can focus on sql only. J. On Thu, Nov 24, 2022 at 4:14 PM Jarek Potiuk wrote: > > Hey all, > > I have just cut the new wave Airflow Providers packages. This email is > calling a vote on the release, > which will

[VOTE] Airflow Providers prepared on November 24, 2022 (RC2)

2022-11-24 Thread Jarek Potiuk
Hey all, I have just cut the new wave Airflow Providers packages. This email is calling a vote on the release, which will last for 72 hours + 1 day for Thanksgiving/ Long Weekend - which means that it will end on Mon 27 Nov 16:15:11 CET 2022. Consider this my (binding) +1. ### DESCRIPTION OF

Re: CVE-2022-40954: Apache Airflow Spark Provider, Apache Airflow: Airflow 2.3.4 spark provider RCE that bypass restrictions to read arbitrary files

2022-11-24 Thread Jarek Potiuk
I've added that in follow-up message: moderate On Thu, Nov 24, 2022 at 12:12 PM Adam Bannister wrote: > > Hi Jarek, > > Any indication what severity tier this bug might be? > > Regards, > Adam > The Daily Swig > > On Mon, 21 Nov 2022 at 21:46, Jarek Po

Re: [VOTE] November 2022 PR of the Month

2022-11-22 Thread Jarek Potiuk
+1 for 26457 too On Tue, Nov 22, 2022 at 5:44 PM Bas Harenslak wrote: > > +1 for 26457 > > Bas > > On 22 Nov 2022, at 17:31, Jeambrun Pierre wrote: > > Hello, > > I vote for https://github.com/apache/airflow/pull/26457. Not sure why it > wasn't selected by our heuristic, with 2k additions,

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

2022-11-22 Thread Jarek Potiuk
e not, so it's may be a > problem that you raised on the topic. Or may be not. It's discussion anyway (: > > -- > ,,,^..^,,, > > > On Tue, Nov 22, 2022 at 3:00 PM Jarek Potiuk wrote: >> >> I propose let's not "diverge" the discussion with this specific case >

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

2022-11-22 Thread Jarek Potiuk
; On 22-Nov-2022 at 4:35:09 PM, Alexander Shorin wrote: >> >> On Tue, Nov 22, 2022 at 1:37 PM Jarek Potiuk wrote: >>> >>> BTW. "Workers from 2.2" used with "Airflow 2.4" is not even a thing. >>> This is something that you should never, ever, try

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

2022-11-22 Thread Jarek Potiuk
ion and reasoning (and finally consensus or voting) rather than letting "code" or "documentation" (which might be wrong) decide on it. "Community over code" in full swing here. J. On Tue, Nov 22, 2022 at 11:28 AM Jarek Potiuk wrote: > > Changing the DB s

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

2022-11-22 Thread Jarek Potiuk
since you cannot > run workers from 2.2.x with a database from 2.4.x. > > -- > ,,,^..^,,, > > > On Tue, Nov 22, 2022 at 12:32 PM Jarek Potiuk wrote: >> >> Hello everyone, >> >> We had a few discussions in PRs recently about removing some >>

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

2022-11-22 Thread Jarek Potiuk
Also for those who prefer visual ways - obligatory XKCD which is the illustration of Hyrum's Law: https://xkcd.com/1172/ On Tue, Nov 22, 2022 at 10:31 AM Jarek Potiuk wrote: > > Hello everyone, > > We had a few discussions in PRs recently about removing some > functionality from t

[DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

2022-11-22 Thread Jarek Potiuk
Hello everyone, We had a few discussions in PRs recently about removing some functionality from the Airflow Core and the question of backwards compatibility came up. Example two discussions: * https://github.com/apache/airflow/pull/27826 * https://github.com/apache/airflow/pull/27067 I think we

[DISCUSSION] Understanding consequences of current Provider version policies

2022-11-21 Thread Jarek Potiuk
Hello Airflow Community, I decided to start a thread (just to avoid some surprises) that will help us to be well aware of some of the consequences of the currently agreed policy we have for providers version support. I think it's good we re-iterate it now. Just to explain my view: I am

Re: CVE-2022-40954: Apache Airflow Spark Provider, Apache Airflow: Airflow 2.3.4 spark provider RCE that bypass restrictions to read arbitrary files

2022-11-21 Thread Jarek Potiuk
Just to add severity: moderate. On Mon, Nov 21, 2022 at 9:41 PM Jarek Potiuk wrote: > > Description: > > Improper Neutralization of Special Elements used in an OS Command ('OS > Command Injection') vulnerability in Apache Airflow Spark Provider, Apache > Airflow allows a

CVE-2022-41131: Apache Airflow Hive Provider vulnerability (command injection via hive_cli connection)

2022-11-21 Thread Jarek Potiuk
Severity: moderate Description: Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection') vulnerability in Apache Airflow Hive Provider, Apache Airflow allows an attacker to execute arbtrary commands in the task execution context, without write access to DAG

CVE-2022-40954: Apache Airflow Spark Provider, Apache Airflow: Airflow 2.3.4 spark provider RCE that bypass restrictions to read arbitrary files

2022-11-21 Thread Jarek Potiuk
Description: Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection') vulnerability in Apache Airflow Spark Provider, Apache Airflow allows an attacker to read arbtrary files in the task execution context, without write access to DAG files. This issue affects

CVE-2022-40189: Apache Airlfow Pig Provider RCE

2022-11-21 Thread Jarek Potiuk
Severity: moderate Description: Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection') vulnerability in Apache Airflow Pig Provider, Apache Airflow allows an attacker to control commands executed in the task execution context, without write access to DAG

CVE-2022-38649: Apache Airflow Pinot Provider, Apache Airflow: PinotAdminHook Command Injection

2022-11-21 Thread Jarek Potiuk
Severity: moderate Description: Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection') vulnerability in Apache Airflow Pinot Provider, Apache Airflow allows an attacker to control commands executed in the task execution context, without write access to DAG

Airflow Providers relesead on 18th of November

2022-11-18 Thread Jarek Potiuk
Dear Airflow community, I'm happy to announce that new versions of Airflow Providers packages were just released. This was a special release - this is the first wave of Airflow 2.3+ only providers - all subsequent release will only be compatible with Airflow 2.3 and you need to update to Airflow

[RESULT][VOTE] Airflow Providers - release prepared November 15, 2022

2022-11-18 Thread Jarek Potiuk
Hello, Apache Airflow Providers (based on RC1 + cncf.kubernetes RC3) have been accepted. We exclude the following providers from this wave (follow up is coming): yandex, asana, jdbc 3 "+1" binding votes received: - Jarek Potiuk (binding) - Ephraim Anierobi (binding) - Daniel Standis

Re: [LAZY CONSENSUS] Do not treat min-airflow-version bump in providers as breaking

2022-11-18 Thread Jarek Potiuk
No opposition. Lazy Consensus agreed. I release providers accordingly. On Mon, Nov 14, 2022 at 10:31 PM Daniel Standish wrote: > > I think it makes sense and I'm a +1. > > For the convenience of other readers I'll paste your rationale here: > >> The rationale i have - that from the point of view

Re: [VOTE] Airflow Providers prepared on November 15, 2022

2022-11-18 Thread Jarek Potiuk
-amazon==6.1.0rc1, >> apache-airflow-providers-apache-hive==4.1.0rc1, >> apache-airflow-providers-apache-livy==3.2.0rc1, >> apache-airflow-providers-common-sql==1.3.0rc1, >> apache-airflow-providers-databricks==3.4.0rc1, >> apache-airflow-providers-google==8.5.0rc1, >>

Re: [VOTE] Airflow Providers prepared on November 15, 2022

2022-11-17 Thread Jarek Potiuk
and google provider worked as expected. >> >> On Thu, Nov 17, 2022 at 7:03 PM Jarek Potiuk wrote: >>> >>> Would love some votes - we have a follow up release after a few small >>> providers docs/fixes, so would be great to release it before tomorrow >>&g

Re: [VOTE] Airflow Providers prepared on November 15, 2022

2022-11-17 Thread Jarek Potiuk
Would love some votes - we have a follow up release after a few small providers docs/fixes, so would be great to release it before tomorrow and start the follow up right after :) On Tue, Nov 15, 2022 at 1:58 AM Jarek Potiuk wrote: > > Hey all, > > I have just cut the new wave Airfl

Re: [VOTE] AIP-51 - Removing Executor Coupling from Core Airlfow

2022-11-15 Thread Jarek Potiuk
+1 (binding) On Tue, Nov 15, 2022 at 10:07 PM Oliveira, Niko wrote: > > Hey folks! > > I would like to start a vote for "AIP-51 - Removing Executor Coupling from > Core Airlfow". > > You can find the AIP here: >

Re: [DISCUSSION] Release of API clients

2022-11-15 Thread Jarek Potiuk
And just to add one "clarifying" condition here: If we release Airflow 2.X.0 (the first release in the MINOR version, the answer to "Do we need to release API Client ?" is always "YES". J. On Tue, Nov 15, 2022 at 12:41 PM Jarek Potiuk wrote: > > &

Re: [DISCUSSION] Release of API clients

2022-11-15 Thread Jarek Potiuk
The logic is simple: * release Airflow * do we need to release Python API Client -> if yes, release * do we need to release Go API Client -> if yes, release * release Docker Image I hope this "algorithm" is pretty straightforward. J. On Tue, Nov 15, 2022 at 12:21 PM Sumit Mah

Re: CVE-2022-27949: Apache Airflow: sensitive values in rendered template

2022-11-15 Thread Jarek Potiuk
Additional info: Credit: Apache Airflow PMC would like to thank James Srinivasan for reporting it. On Mon, Nov 14, 2022 at 12:50 AM Jarek Potiuk wrote: > > Severity: low > > Description: > > A vulnerability in UI of Apache Airflow allows an attacker to view unmasked >

Re: [DISCUSSION] Release of API clients

2022-11-15 Thread Jarek Potiuk
> > There had been a discussion around this earlier, though I couldn't find the > thread. So if we want to release clients with each Airflow release, then > we've to move away from current semantic versioning, something which we had > decided in that thread. Why ? I think we do not have to do

[VOTE] Airflow Providers prepared on November 15, 2022

2022-11-14 Thread Jarek Potiuk
Hey all, I have just cut the new wave Airflow Providers packages. This email is calling a vote on the release, which will last for 72 hours - which means that it will end on Friday, November 18, 2022 2 am CET. Consider this my (binding) +1. This is a special release as all providers are only

Re: [DISCUSSION] Release of API clients

2022-11-14 Thread Jarek Potiuk
+1 on adding it to the release process. There is even a very good reason for it, which will become all but mandatory. In the future, when we implement Internal API AIP (AIP-44) - if the user will want to use some airflow feature, they should be able to use the Python API client to communicate

Re: [Discussion] Airflow Newsletter name and branding

2022-11-14 Thread Jarek Potiuk
Love the "Atmosphere" name :) +1 On Mon, Nov 14, 2022 at 9:09 PM Mehta, Shubham wrote: > +1 (non-binding) to the “Atmosphere: The Airflow Monthly” > > > > Other suggestion - “Symphony: The Airflow Monthly” > Context – In one of the talks Jarek compared Airflow to orchestra who > guides tasks

Re: [LAZY CONSENSUS] Do not treat min-airflow-version bump in providers as breaking

2022-11-14 Thread Jarek Potiuk
11, 2022 at 2:28 PM Jarek Potiuk wrote: > > Hey everyone, > > Following the explanation in > https://lists.apache.org/thread/xpngxsdxmk1vw2wk34py8sdsqfmjdw9g I would > like to call for a lazy consensus on the change proposed in the PR > https://github.com/apac

Re: November 2022 wave of providers

2022-11-14 Thread Jarek Potiuk
* cncf.kubernetes * microsoft-azure * opsgenie * tableau * slack * snowflake PR - ready to approve https://github.com/apache/airflow/pull/27613 - looking forward to this one. J. On Fri, Nov 11, 2022 at 2:21 PM Jarek Potiuk wrote: > > YyFew more updates: > > 1) I went

Re: [VOTE] Release Airflow 2.4.3 from 2.4.3rc1

2022-11-13 Thread Jarek Potiuk
+1 (binding) - checked signatures, checksums, licences, run Airflow and a number of DAGs, all looks good. On Fri, Nov 11, 2022 at 2:31 PM Ephraim Anierobi wrote: > > Hey fellow Airflowers, > > > I have cut Airflow 2.4.3rc1. This email is calling a vote on the release, > > which will last at

CVE-2022-27949: Apache Airflow: sensitive values in rendered template

2022-11-13 Thread Jarek Potiuk
Severity: low Description: A vulnerability in UI of Apache Airflow allows an attacker to view unmasked secrets in rendered template values for tasks which were not executed (for example when they were depending on past and previous instances of the task failed). This issue affects Apache

CVE-2022-40127: RCE in Apache Airflow <2.4.0 bash example

2022-11-13 Thread Jarek Potiuk
Severity: low Description: A vulnerability in Example Dags of Apache Airflow allows an attacker with UI access who can trigger DAGs, to execute arbitrary commands via manually provided run_id parameter. This issue affects Apache Airflow Apache Airflow versions prior to 2.4.0. Mitigation:

Re: [ANNOUNCEMENT] Airflow CI Infra news

2022-11-11 Thread Jarek Potiuk
ar bigger users). > > > Do you have a link to this attachment? Thanks! > > Best, > Solomon > > > On Fri, Nov 11, 2022 at 1:17 AM Jarek Potiuk wrote: > >> Hello Everyone, >> >> I thought it's a good time to share some news after yesterday's ASF >&g

<    3   4   5   6   7   8   9   10   11   12   >