Re: [VOTE] November PR of the Month

2023-11-27 Thread Jarek Potiuk
+ 1 for 32646 On Mon, Nov 27, 2023 at 10:35 PM utkarsh sharma wrote: > +1 for 32646, great work. > > Thanks, > Utkarsh Sharma > > On Tue, Nov 28, 2023 at 2:50 AM Collin McNulty > > wrote: > > > +1 to 32646. Been wanting this for a long time! > > > > On Mon, Nov 27, 2023 at 11:50 AM Briana

Re: [VOTE] Airflow Providers prepared on November 24, 2023

2023-11-26 Thread Jarek Potiuk
+1 (binding) Tested my template changes, signatures, checksum, licences, sources are all good. I also tested that reproducible builds work great ! I checked out the tag from sources, ran `breeze release-management prepare-provider-packages` and got the packages built locally which had 100%

[LAZY CONSENSUS] Remove 'daskexecutor' provider

2023-11-24 Thread Jarek Potiuk
Hello everyone, This is a formal call for consensus to remove the daskexecutor provider. All the details and proposal here https://lists.apache.org/thread/ptwjf5g87lyl5476krt91bzfrm96pnb1 I believe we can remove it straight away - my announcement in https://github.com/dask/community/issues/355

Re: [DISCUSS] Suspend/Remove Apache Scoop provider

2023-11-24 Thread Jarek Potiuk
Also - with all these discussions we have to remember that we are NOT forbidding anyone to use Scoop or any other provider that is out there. They can continue using it for as long they can install released latest version of the provider. Which means basically forever (or as long as dependencies

Re: [DISCUSS] Suspend/Remove Plexus provider

2023-11-23 Thread Jarek Potiuk
Let's remove it. On Thu, Nov 23, 2023 at 11:52 PM Andrey Anshin wrote: > Greetings everyone! > > Not much time has passed since the last discussion about suspension/removal > providers 藍 > > It is time to discuss Plexus Provider. > > During check providers links, I've found that link [1] from

Re: [DISCUSS] Suspend/Remove Apache Scoop provider

2023-11-23 Thread Jarek Potiuk
+1 czw., 23 lis 2023, 12:39 użytkownik Aritra Basu napisał: > +1 sounds like a good reason to suspend it and eventually remove it. > > -- > Regards, > Aritra Basu > > On Thu, Nov 23, 2023, 4:45 PM Bolke de Bruin wrote: > > > +1 > > > > Go > > > > Sent from my iPhone > > > > > On 23 Nov 2023,

[LAZY CONSENSUS] Change default docker image to point to "latest supported"

2023-11-21 Thread Jarek Potiuk
Hello everyone, Following the proposal https://lists.apache.org/thread/xy192wmv6k1sy2wlx5vy64p678skrhko I would like to ask for a Lazy consensus on making default docker image point to "latest supported" version (so for 2.8.0 it would be 3.11). Reasoning and discussion available in the PROPOSAL

Re: [LAZY CONSENSUS] Add fab provider

2023-11-21 Thread Jarek Potiuk
Oh Fully. agre. It's a bit too late to do it for 2.8 :) . But yeah no objections here :) On Wed, Nov 22, 2023 at 12:45 AM Kaxil Naik wrote: > No objection, let's include it for 2.9 so we can test it thoroughly. > > On Tue, 21 Nov 2023 at 15:05, Beck, Vincent > wrote: > > > Hello everyone, > >

Re: [DISCUSS] Future of Pendulum in Airflow

2023-11-21 Thread Jarek Potiuk
uld we do in this case? > - Work out a backup plan. > > > > Best Wishes > *Andrey Anshin* > > > > On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk wrote: > > > Also I think TP - had a document in the past (years ago) describing a > > draft of a more com

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

2023-11-20 Thread Jarek Potiuk
816805948> > > On Fri, 17 Nov 2023 at 17:06, Jarek Potiuk wrote: > > > I was asked to open the issue in GitHub to get more visibility by Dask > > developers so here it is https://github.com/dask/community/issues/355 > > > > On Fri, Nov 17, 2023 at 1:21 PM Jarek Potiu

Re: [PROPOSAL] Deprecate URI Connection representation in favor of JSON

2023-11-18 Thread Jarek Potiuk
I agree with Ash that it's much easier to define a lot of simpler connections this way. But it is also very confusing the way now how you can mix the "real" url with "airflow connection" URL. And Daniel is very right about the magnitude of breaking change. But possibly there is a way to eat

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

2023-11-17 Thread Jarek Potiuk
I was asked to open the issue in GitHub to get more visibility by Dask developers so here it is https://github.com/dask/community/issues/355 On Fri, Nov 17, 2023 at 1:21 PM Jarek Potiuk wrote: > OK. Seeing that - I think I will do the next step - I pointed this > discussion to at the d

Re: [DISCUSS] Airflow UI DAG Composer

2023-11-17 Thread Jarek Potiuk
I added you, Yulei, I have no time to look at details, but I have two big concerns about this - regarding Audience and Security (first concern) and whether we want to do it all (second concern). First about security and audience: This is against the current Security Model of Airflow:

Re: [PrOPOSAL] Change default docker image to point to "latest supported"

2023-11-17 Thread Jarek Potiuk
y or logic-intensive tasks and CPU usage is high for airflow workers, not only scheduler - and in those cases those performance improvements will be fairly visible. On Fri, Nov 17, 2023 at 1:45 PM Jarek Potiuk wrote: > Yeah. I see the point of Andrey - indeed, we had - for quite some time -

Re: [PrOPOSAL] Change default docker image to point to "latest supported"

2023-11-17 Thread Jarek Potiuk
t; Aritra Basu > > > > > > On Fri, Nov 17, 2023, 12:33 AM Vincent Beck > wrote: > > > > > >> I agree, by default we should use the latest python version. Like any > > >> package manager, if the user does not explicitly specify

Re: [DISCUSS] Future of Pendulum in Airflow

2023-11-17 Thread Jarek Potiuk
Also I think TP - had a document in the past (years ago) describing a draft of a more complete alternative we can take to approach datetime vs. pendulum dichotomy. I cannot easily find the document and discussion - but I do remember it was proposing some interesting changes in the approach of

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

2023-11-17 Thread Jarek Potiuk
gt;>>>> Daskexecutor provider > >>>>> > >>>>> CAUTION: This email originated from outside of the organization. Do > >> not > >>>>> click links or open attachments unless you can confirm the sender and > >>>&g

Re: [DISCUSS] Move FAB auth manager to a new provider

2023-11-17 Thread Jarek Potiuk
Ah ... put it in a wrong thread, sorry :) ... On Fri, Nov 17, 2023 at 12:39 PM Jarek Potiuk wrote: > OK. Seeing that - I think I will do the next step - I will point this > discussion to at the discord of Dask and see if there is a volunteer there > who would like to take

Re: [DISCUSS] Move FAB auth manager to a new provider

2023-11-17 Thread Jarek Potiuk
17, 2023 at 10:28 AM Wei Lee wrote: > +1 for this moving it. It gives us more flexibility on both the core and > provider sides. > > Best, > Wei > > > On Nov 17, 2023, at 9:15 AM, Jarek Potiuk wrote: > > > > I am all for it. As we saw already and we see it more in

Re: [DISCUSS] Move FAB auth manager to a new provider

2023-11-16 Thread Jarek Potiuk
I am all for it. As we saw already and we see it more in the future - moving code of out of Airflow core to provider and having separate provider's release cycle and lifecycle is generally beneficial: * dependencies can be more decoupled - even if we pin FAB with a particular version of provider,

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

2023-11-16 Thread Jarek Potiuk
would go for immediate removal. WDYT? J. On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk wrote: > > On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif wrote: > >> Now that the code is in its own provider we can check the download stats >> of >> the library via Pyp

[PrOPOSAL] Change default docker image to point to "latest supported"

2023-11-16 Thread Jarek Potiuk
Hello everyone, Since we are close to the Airflow 2.8.0 release, I would like to propose a change in the approach for our "default" images. Currently there are few images that are considered as "default", for example: apache/airflow:latest apache/airflow:2.7.4 Currently (according to our

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

2023-11-15 Thread Jarek Potiuk
On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif wrote: > Now that the code is in its own provider we can check the download stats of > the library via Pypi stats. > > Good point but this is only an indication. Currently this is only for Airflow 2.7+ and it is a bit difficult to compare those. The

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

2023-11-15 Thread Jarek Potiuk
cing its suspension? > > On Wed, Nov 15, 2023 at 3:32 PM Jarek Potiuk wrote: > >> Hello Everyone, >> >> I would like to start discussion about potential suspension and eventually >> removal of Daskexecutor provider. >> >> After some recent changes and moving

[DISCUSS] Suspend (Remove?) Daskexecutor provider

2023-11-15 Thread Jarek Potiuk
Hello Everyone, I would like to start discussion about potential suspension and eventually removal of Daskexecutor provider. After some recent changes and moving DaskExecutor from core to provider we have now an open option to suspend and eventually remove Dask Executor from our codebase. After

Re: [LAZY CONSENSUS] Remove Qubole provider

2023-11-14 Thread Jarek Potiuk
repository. We will only potentially release new versions if we decide to fix a security issue reported to us, however since the service is not maintained any more, it's highly unlikely we will decide to do it. J. On Tue, Nov 7, 2023 at 1:56 PM Jarek Potiuk wrote: > The "technical" s

Re: [VOTE] Airflow Providers prepared on November 12, 2023

2023-11-13 Thread Jarek Potiuk
+1 (binding): checked signatures, checksums, licences, sources. Looks like the fixes were also already well tested in https://github.com/apache/airflow/issues/35592 On Sun, Nov 12, 2023 at 10:43 PM Elad Kalif wrote: > Hey all, > > I have just cut the RC2 wave Airflow Providers packages. This

Re: [VOTE] Release Airflow Python Client 2.7.3 from 2.7.3rc1

2023-11-13 Thread Jarek Potiuk
Sorry for missing it for so long. +1 (binding). Checked licences, checksums, signatures, tested that it works with latest airflow - following https://github.com/apache/airflow-client-python/blob/main/dev/test_python_client.py I also verified sources (finally). One of the things that we do as

Re: [VOTE] Airflow Providers prepared on November 08, 2023

2023-11-11 Thread Jarek Potiuk
3 at 13:42, Kaxil Naik wrote: > > > > > >> -1 (binding) on common.io Provider > > >> https://pypi.org/project/apache-airflow-providers-common-io/1.0.1rc1/ > > >> > > >> It isn't going to work with Airflow 2.7.*, it needs code that is main >

Re: [VOTE] Airflow Providers prepared on November 08, 2023

2023-11-11 Thread Jarek Potiuk
One thing to add to my +1. I have not mentioned it before, but I promised to the team contributing the new LLM providers to review the double check licences / dependencies of our new LLM providers. Just to confirm, it all looks good. We do not have a strict requirement to adhere to a permissive

Re: [VOTE] Airflow Providers prepared on November 08, 2023

2023-11-08 Thread Jarek Potiuk
+1 (binding) - checked sources, signatures, checksums, licences. I had just a little change about verifying the structure of provider docs. I also checked that the scheduled-to-remove qubole provider last release contains the "removal warning" :

[ANNOUNCE] New committer: Jens Scheffler

2023-11-07 Thread Jarek Potiuk
The Project Management Committee (PMC) for Apache Airflow has invited Jens Scheffler to become a committer and we are pleased to announce that they have accepted. Jens has been contributing for a number of months he also participated a lot in discussions and decisions on many aspects of Airflow

Initiative on Strengthenig security for Apache Airflow

2023-11-07 Thread Jarek Potiuk
Hello everyone, I wanted to share some news (not so much news for us but - it's just now reached publication stage) that we have nice security / release process improvements on-going in Apache Airflow - with several months of work funded by the Sovereign Tech Fund - German government backed fund

Re: [LAZY CONSENSUS] Remove Qubole provider

2023-11-07 Thread Jarek Potiuk
Also I consolidated "technical" howtos on how to create, suspend, resume, remove providers into a single MANAGING_PROVIDERS_LIFECYCLE document where the technical aspects of all those are explained. J. On Mon, Nov 6, 2023 at 10:39 AM Jarek Potiuk wrote: > The lazy consensus has been re

Re: [LAZY CONSENSUS] Migrating our reference images to Debian Bookworm for Airflow 2.8

2023-11-06 Thread Jarek Potiuk
n https://github.com/apache/airflow/pull/35487 J. On Mon, Nov 6, 2023 at 10:32 AM Jarek Potiuk wrote: > The lazy consensus has been reached. I will proceed with merging the > https://github.com/apache/airflow/pull/35376 > > THanks again Raphael for the reminders and being persistent

Re: [LAZY CONSENSUS] Remove Qubole provider

2023-11-06 Thread Jarek Potiuk
The lazy consensus has been reached. I will proceed with Qubole removal following the new process we have in place: https://github.com/apache/airflow/blob/main/PROVIDERS.rst#removing-community-providers On Mon, Oct 30, 2023 at 7:24 PM Jarek Potiuk wrote: > Hello everyone, > > As

Re: [LAZY CONSENSUS] Agree on the proces of removal of providers

2023-11-06 Thread Jarek Potiuk
Lazy consensus has been reached. I am proceeding with merging https://github.com/apache/airflow/pull/35277 for the process change. On Tue, Oct 31, 2023 at 8:33 PM Jarek Potiuk wrote: > Hello everyone, > > We seem to have general consensus about the need and general approach for &

Re: [LAZY CONSENSUS] Migrating our reference images to Debian Bookworm for Airflow 2.8

2023-11-06 Thread Jarek Potiuk
The lazy consensus has been reached. I will proceed with merging the https://github.com/apache/airflow/pull/35376 THanks again Raphael for the reminders and being persistent :) On Thu, Nov 2, 2023 at 8:46 PM Jarek Potiuk wrote: > Hello everyone, > > *TL;DR;* Following our OS upgrade

Re: [ANNOUNCE] Apache Airflow 2.7.3 Released

2023-11-05 Thread Jarek Potiuk
wooohooo! On Mon, Nov 6, 2023 at 7:58 AM Ephraim Anierobi wrote: > Dear Airflow community, > > I'm happy to announce that Airflow 2.7.3 was just released. > > The released sources and packages can be downloaded via >

Re: [Reminder] How to reproducibly install Airflow

2023-11-05 Thread Jarek Potiuk
s is also recommended as well > right? (unless I'm wrong, an official Airflow Docker image uses pip for > installing Airflow, including also the constraints file for the > dedicated version). > > Here we've been using Docker compose installation for years without > encounter any m

Re: [Reminder] How to reproducibly install Airflow

2023-11-04 Thread Jarek Potiuk
t by pin third meta issue into the > Github Issues which are described best practices include: > - Reproducible install > - What we expect of good bug/feature request > - Information about third-party Managed Airflow > > > Best Wishes > *Andrey Anshin* > &

[Reminder] How to reproducibly install Airflow

2023-11-04 Thread Jarek Potiuk
Dear Airflow community, Since there were a number (more than 4 already) of issues opened recently when Connexion 3 broke installation of the released Airflow version a few days ago - I have a short reminder on how to install Airflow in a reproducible way. If you want to make sure released

Re: [VOTE] Release Airflow 2.7.3 from 2.7.3rc1

2023-11-02 Thread Jarek Potiuk
+1 (binding) Checked checksums, signatures, licences and sources. All looks good. Ran a few DAGs, clicked through UI. I run airflow with Local/Celery executors - it looks good. Found that the image's `pip` version is not what was expected (and was always lagging a bit) but this is not a blocker

[LAZY CONSENSUS] Migrating our reference images to Debian Bookworm for Airflow 2.8

2023-11-02 Thread Jarek Potiuk
Hello everyone, *TL;DR;* Following our OS upgrade policy [1] - I ask for a lazy consensus to switch our Docker images from Bullseye to Bookworm. The 2.8 version will be based on Bookworm, and we keep an option to build a custom Bullseye image for users who need it). In 2.9 we will drop Bullseye

Re: [DISCUSS] API Clients Major version

2023-11-02 Thread Jarek Potiuk
pose to go for a breaking.change and accept that > > the client‘s version number is not in sync with Airflow anymore. > > > > Also this gives the opportunity to maintain a branch of the old if needed > > and still keep the same name. > > > > Jens > > > > S

Re: [DISCUSS] API Clients Major version

2023-11-01 Thread Jarek Potiuk
Alternative proposal. Why don't we release a new package 'python-client-nextgen' & with new build process/generator and release process following the same versioning as previously.. The old python client will continue to work for current APIa but it will stop receiving new features so if

[ANNOUCEMENT] HOPEFULLY BIG Speed/Stability improvements of CI by DB / NonDB test splitting merged

2023-10-31 Thread Jarek Potiuk
Hello everyone, *TL;DR; The ew shiny and likely much snappier test harness for unit tests is merged to main.* *REQUEST: I have a kind request to committers and contributors - please remember to rebase all the PRs you are merging in the next few days as some of those PRs might break main if not

[LAZY CONSENSUS] Agree on the proces of removal of providers

2023-10-31 Thread Jarek Potiuk
Hello everyone, We seem to have general consensus about the need and general approach for removing providers. We had discussion about it in https://lists.apache.org/thread/x4gt2h5hql7j04jj0v7v7kzzv1nkrzxy with generally everyone being in favour of both - removing Qubole and capturing the removal

Re: [VOTE] October 2023 PR of the Month

2023-10-31 Thread Jarek Potiuk
+1 to 34729 as well. On Tue, Oct 31, 2023 at 6:38 PM Constance Martineau wrote: > Oh, I'm very sorry. I had forgotten about 34729. Apologies, but I'll be > changing my vote to that. > > On Tue, Oct 31, 2023 at 1:08 PM Hussein Awala wrote: > > > I vote for 34729

[LAZY CONSENSUS] Remove Qubole provider

2023-10-30 Thread Jarek Potiuk
Hello everyone, As discussed in https://lists.apache.org/thread/x4gt2h5hql7j04jj0v7v7kzzv1nkrzxy this email clls for lazy consensus on removal of the Qubole provider. In short - we want to remove the Qubole provider as the company has been acquired and the service and clients used to

Re: [DISCUSS] Removing Qubole provider (and adding removal process)

2023-10-30 Thread Jarek Potiuk
; > >>> Sounds like a good time to set the process up. +1 from me as well. > > >>> > > >>> -- > > >>> Regards, > > >>> Aritra Basu > > >>> > > >>> On Fri, Oct 27, 2023, 6:42 PM Vincent Beck > &g

Re: [VOTE] Add Multiple PR's of the Month to the Airflow Newsletter

2023-10-30 Thread Jarek Potiuk
It was Jed's comment, but yes. I agree with it too :) On Mon, Oct 30, 2023 at 5:51 PM Briana Okyere wrote: > Thank you Jarek, this is very helpful. > > With your input in mind, let's move forward with a more flexible structure. > If there is no clear Top PR, we can feature multiple. If there

Re: Changing configuration options not updating airflow webserver login options

2023-10-30 Thread Jarek Potiuk
Kind request: please use dedicated channels for this kind of questions (users list, slack, Github Discussions): https://airflow.apache.org/community/ links ("ask a question", "start a discussion", "user list" ). Devlist is mostly for development related discussions. J, On Mon, Oct 30, 2023 at

Re: Airflow Docs Development Issues

2023-10-29 Thread Jarek Potiuk
> > > > Just one thought maybe we can enforce the process to achieve docs, maybe > via pre-commit hooks/updating the `breeze release-management publish-docs` > command. So that anytime there is something new published we also check the > docs to achieve . > Yep. That would be nice to archive the

Re: [PROPOSAL] Explain what milestones mean in our PRs/Issues - what users can expect

2023-10-29 Thread Jarek Potiuk
nd pointing to the spec could be > > helpful. > > > > Le sam. 28 oct. 2023 à 22:14, Jarek Potiuk a écrit : > > > >> Hey everyone, > >> > >> We had some discussions in the past about codifying the approach we have > >> for the Github Is

Re: [PROPOSAL] (likely) significantly shorten CI test time: splitting to db/non-db tests

2023-10-29 Thread Jarek Potiuk
provements on Public Runners. And stability seems to be much better now compared to the problems we had recently with timeouts and some tests taking longer while competing with other parallel tests. At least for now 爛 On Sun, Oct 29, 2023 at 5:09 PM Jarek Potiuk wrote: > I got the first &quo

Re: [PROPOSAL] (likely) significantly shorten CI test time: splitting to db/non-db tests

2023-10-29 Thread Jarek Potiuk
I got the first "fully green pass" of the "Improve testing harness to separate DB and non-DB tests" that looks stable and shows the "real" numbers and improvements. Copying it here as well from https://github.com/apache/airflow/pull/35160#issuecomment-1784152557 as this one will impact (I hope

Re: [VOTE] Airflow Providers prepared on October 28, 2023

2023-10-28 Thread Jarek Potiuk
+1 (binding): checked my change in presto. Checked licences, signatures, checksums, sources. All looks good. On Sat, Oct 28, 2023 at 9:20 PM utkarsh sharma wrote: > Got it, thanks Elad and Jarek. :) > > > > On Sun, Oct 29, 2023 at 12:43 AM Jarek Potiuk wrote: &

[PROPOSAL] Explain what milestones mean in our PRs/Issues - what users can expect

2023-10-28 Thread Jarek Potiuk
Hey everyone, We had some discussions in the past about codifying the approach we have for the Github Issues and PRs of ours / Milestones - so that users can know what to expect. We have - I believe - pretty good understanding of it amongst those who are involved in the release management but I

Re: [VOTE] Airflow Providers prepared on October 28, 2023

2023-10-28 Thread Jarek Potiuk
And to add to it: the "status" issue https://github.com/apache/airflow/issues/35240 serves largely as the "pre-release" changelog. On Sat, Oct 28, 2023 at 9:11 PM Elad Kalif wrote: > docs are published only after the vote is accepted and we release the > packages. > PR

Re: [VOTE] Add Multiple PR's of the Month to the Airflow Newsletter

2023-10-27 Thread Jarek Potiuk
Yeah. Top PRs of the month sound good. On Fri, Oct 27, 2023 at 6:16 PM Amogh Desai wrote: > Hi, > > I also think having multiple PRs under PR of the month would be really > nice. > > One way to approach this is: > > What we can do is, collect votes for all the stakeholders for their top 3 > PRs

Re: [DISCUSS] Removing Qubole provider (and adding removal process)

2023-10-27 Thread Jarek Potiuk
takeholders (happened with Yandex - both suspend and resume) * we (will finally) know how to retire them when we decide we do not want to maintain them - except security fixes - any more That will pretty much complete our process of "life-cycle" management for providers. J. On Thu, O

Re: Airflow Docs Development Issues

2023-10-27 Thread Jarek Potiuk
; > Yup, sounds good to me let's go for it! > > > > > > > > -- > > > > Regards, > > > > Aritra Basu > > > > > > > > On Thu, Oct 26, 2023, 1:47 PM Amogh Desai > > > > wrote: > > > > > > > >

Re: [DISCUSS] Removing Qubole provider (and adding removal process)

2023-10-26 Thread Jarek Potiuk
ely manner). The act of release with 3 +1s of PMC is a legal act of the Foundation placing software on the market and we can't make it "unhappen". > B. > > Sent from my iPhone > > > On 26 Oct 2023, at 20:20, Jarek Potiuk wrote: > > > > Hello Airflow c

[DISCUSS] Removing Qubole provider (and adding removal process)

2023-10-26 Thread Jarek Potiuk
Hello Airflow community, How do we feel about removing the Qubole provider completely (leaving only old releases in PyPI? On September 1 2023 ( https://lists.apache.org/thread/p394d7w7gc7lz61g7qdthl96bc9kprxh) the Qubole operator ws suspended. Due to the reasons described in the thread (Qubole

Re: [PROPOSAL] (likely) significantly shorten CI test time: splitting to db/non-db tests

2023-10-26 Thread Jarek Potiuk
BTW. I also plan to add short "best practices" chapter to our TESTING.rst to deal with some of those "special" cases based on the learning from that whole experience. On Thu, Oct 26, 2023 at 3:46 PM Jarek Potiuk wrote: > All right. > > I think I am getti

Re: [PROPOSAL] (likely) significantly shorten CI test time: splitting to db/non-db tests

2023-10-26 Thread Jarek Potiuk
ng this decorator, > > mostly because I think the effort is not huge. I also think, as a > > contributor, it is a good exercise to, when writing tests, figure out > > whether the tests I am writing is using the DB. > > > > On 2023/10/24 16:31:57 Jarek Potiuk wrote: > > > Hello

Re: Limiting (or errorring out) Airflow for Python 3.12 until our dependencies/we catch up

2023-10-26 Thread Jarek Potiuk
ng an official "soft depreciation", > > which will avoid the eventual removal of a function but no longer support > > any development: > > > https://discuss.python.org/t/formalize-the-concept-of-soft-deprecation-dont-schedule-removal-in-pep-387-backwards-compatibility-

Re: [Discussion] Requesting Feedback on Airflow Survey

2023-10-25 Thread Jarek Potiuk
Agree with Andrey's suggestions - added a few of mine directly to the docs as comments/suggestions. Summary of my comments: * mentioning 2.5 providers compatibility and reasoning why people staying below 2.5 * I thing suggesting list of services/tools Airflow might interact with to choose, will

Re: Keep Mssql support

2023-10-25 Thread Jarek Potiuk
at least a DB scheme upgrade supported for MsSQL in > > order to be able to migrade to a new DB engine in 2.8.0 w/o need of > > transformation of structures for people being stuck on 2.7.3 wirh MsSQL. > > > > Sent from Outlook for iOS<https://aka.ms/o0ukef> > &

Re: Airflow Docs Development Issues

2023-10-25 Thread Jarek Potiuk
e them among > Aritra Basu, Amogh Desai, and myself. > > Thanks, > Utkarsh Sharma > > On Tue, Oct 24, 2023 at 10:12 PM Jarek Potiuk wrote: > > > Those look like great ideas. > > > > On Tue, Oct 24, 2023 at 4:23 PM utkarsh sharma > > wrote: > > > &

Re: Keep Mssql support

2023-10-25 Thread Jarek Potiuk
> The alternatives suggested by @Jarek Potiuk are something which is doable. We need to realise that these plugins are for the community and we can only support it if "majority" of the community uses it and is willing to maintain it :) Just to clarify - it's not about plugins, it'

Re: [VOTE] Add providers for Pinecone, OpenAI & Cohere to enable first-class LLMOps

2023-10-25 Thread Jarek Potiuk
+1 (binding) On Wed, Oct 25, 2023 at 12:25 PM Kaxil Naik wrote: > Hello everyone, > > Following the discussion about adding five new providers, I am calling for > an official vote on adding providers for Pinecone, OpenAI & Cohere to the > Airflow repo. > > Discussion thread: >

Re: Limiting (or errorring out) Airflow for Python 3.12 until our dependencies/we catch up

2023-10-25 Thread Jarek Potiuk
that would > > understand the implication that neither does 2.7.2 and thus they > > wouldn't try installing it. > > > > Though I would also think they would have the same understanding that > > if > > 2.7.3 doesn't > > list 3.12 as supported neither would 2.7

Re: Airflow Docs Development Issues

2023-10-24 Thread Jarek Potiuk
e, right now if you don't know the flow it's a bit > difficult > > to pinpoint the code to change. > > 1. If we want to make changes to a specific provider's content we can do > > it Airflow's repo docs//*.rst file. > > 2. If we have a change that affects multiple providers or

[PROPOSAL] (likely) significantly shorten CI test time: splitting to db/non-db tests

2023-10-24 Thread Jarek Potiuk
Hello everyone, *TL;DR; *I have a proposal on how we can probably significantly decrease time needed to run our CI tests. All that at the expense of small - I think- effort by the need to mark tests as "db-tests" by contributors enforced by our CI (in most cases that won't even be needed).. *A

Re: Airflow Docs Development Issues

2023-10-24 Thread Jarek Potiuk
e to help, I'm not too experienced with this area so I might > > not be able to actually propose what changes need doing, but if someone > has > > a path forward on this I can definitely contribute some time to help out > > given some guidance on what is needed. > > > &

Re: [VOTE] AIP-58 Airflow ObjectStore

2023-10-24 Thread Jarek Potiuk
> > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263430565#AIP58AirflowObjectStore(AS)-Whyisitneeded > > > > > ?> > > > > > : > > > > > > > > > >1. Simplify DAG CI/CD > > &

Limiting (or errorring out) Airflow for Python 3.12 until our dependencies/we catch up

2023-10-23 Thread Jarek Potiuk
Hey everyone, I've opened a PR https://github.com/apache/airflow/pull/35123 to limit Airflow to Python < 3.12 though I am not sure if this is the best idea so I seek devlist wisdom to decide whether we should do this, or maybe something else like allowing airflow to be installed but produce a

Re: Airflow Docs Development Issues

2023-10-22 Thread Jarek Potiuk
t would bring for airflow-site contributions. > > Le jeu. 19 oct. 2023 à 16:11, Jarek Potiuk a écrit : > > > Let me just clarify (because that could be unclear) what my +1 was about. > > > > I was not talking (and I believe Ryan was not talking either) about > >

Re: [VOTE] AIP-58 Airflow ObjectStore

2023-10-19 Thread Jarek Potiuk
that it was suggested > that we might want to > move a bit faster than core on the very simple (yet powerful ;-) ) > FileTransferOperator. > > Considering this I hope you would like to make your measly +1 into a strong > +1 :-). > > Cheers > Bolke > > > On Thu, 19 Oct 202

Re: [VOTE] AIP-58 Airflow ObjectStore

2023-10-19 Thread Jarek Potiuk
Finally caught up with this one, looked through code and discussions. I am a little torn on that one but I did some more research and I think it's a useful abstraction. +1(binding) The big + of using fsspec is that it is already supported by the most important "consumers" that are likely to be

Re: The "no_status" state

2023-10-19 Thread Jarek Potiuk
I think it will be tricky to get all the reasons surfaced to the user why the task is not run. But surfacing it to the user is indeed a good idea. Currently this is only done by this FAQ response - showing possible reasons

Re: Airflow Docs Development Issues

2023-10-19 Thread Jarek Potiuk
ery provider and every version of that provider. If we have a more > dynamic way(Django/Flask Servers) of catering the documents we can save all > the space for common HTML/CSS/JS. > > But the downsides of this approach are: > 1. We need to have a server > 2. Also require changes

Re: Airflow Docs Development Issues

2023-10-19 Thread Jarek Potiuk
Yes. Moving the old version to somewhere that we can keep/archive static historical versions of those historical docs and publish them from there. What you proposed is exactly the solution I thought might be best as well. It would be a great task to contribute to the stability of our docs

Re: [VOTE] Airflow Providers prepared on October 18, 2023

2023-10-18 Thread Jarek Potiuk
+1 (binding). Checked signatures, checksum, licences, source code. On Wed, Oct 18, 2023 at 8:03 PM Elad Kalif wrote: > Hey all, > > I have just cut the new wave Airflow Providers packages. This email is > calling a vote on the release, > which will last for 72 hours - which means that it will

Re: [DISCUSSION] Add 5 new Providers to enable first-class LLMOps

2023-10-18 Thread Jarek Potiuk
I thought a bit about it, and I think the way we have "Astronomer" behind it, it checks all the boxes - providing that we will also have some (super simple) dashboard similar to the MWAA one https://aws-mwaa.github.io/open-source/system-tests/dashboard.html . From

Re: [VOTE] Airflow Providers prepared on October 13, 2023

2023-10-15 Thread Jarek Potiuk
+1 (binding). Tested licences, signatures, checksums, source code. On Sun, Oct 15, 2023 at 2:27 PM Amogh Desai wrote: > +1 (non binding) > > Tested out some DAGs, things seem ok > > Thanks & Regards, > Amogh Desai > > On Sat, Oct 14, 2023, 07:53 Wei Lee wrote: > > > +1 (non-binding) > > > >

Re: Summary of the In-person informal workshops we had in Toronto

2023-09-22 Thread Jarek Potiuk
in an RV in the middle of wild Eeast Canada, without the intention of checking my github or slack or email messages :) On Fri, Sep 22, 2023 at 11:58 AM Jarek Potiuk wrote: > Hey everyone, > > I would like to share a short information about a little informal workshop > we hel

Re: AIP access

2023-09-20 Thread Jarek Potiuk
Added. On Wed, Sep 20, 2023 at 12:29 PM Bartosz Jankiewicz < bartosz.jankiew...@gmail.com> wrote: > My CWiki username is *bartosz.jankiewicz*. > > Best, > Bartosz > > On Tue, 19 Sept 2023 at 20:57, Bartosz Jankiewicz < > bartosz.jankiew...@gmail.com> wrote: > > > Hi, > > > > Can you please grant

[ANNOUNCE] Pankaj Koti and Amogh Desai as committers

2023-09-19 Thread Jarek Potiuk
Hello Everyone: I am happy to announce that the Project Management Committee (PMC) for Apache Airflow has invited Pankaj Koti and Amogh Desai to become committers and we are pleased to announce that they have accepted. Congratulations and welcome aboard! Regards, Jarek on behalf of Airflow PMC

Re: Looking for public good/positive impact projects for Airflow

2023-09-17 Thread Jarek Potiuk
Just to clarify: this is about "if you can talk about a public project that uses Airflow and made a positive impact" - please contact press@ :). They want to hear about it and make a campaign maybe. On Sun, Sep 17, 2023 at 8:56 AM Jarek Potiuk wrote: > Hello Everyone, > >

Looking for public good/positive impact projects for Airflow

2023-09-17 Thread Jarek Potiuk
Hello Everyone, The ASF is looking for projects that have a positive impact on the world that Airflow helps with. Non-profit, innovative, creative, public impact... any kind of positive impact our projects might have would be of interest for M If you have something along these lines where

Re: [VOTE] Airflow Providers prepared on September 14, 2023

2023-09-14 Thread Jarek Potiuk
+1 (binding): checked signatures, checksums, licences, sources match tags. My fix to cncf.kubernetes with misplaced imports for older Airflow versions is there. On Thu, Sep 14, 2023 at 10:59 AM Elad Kalif wrote: > Hey all, > > I have just cut the new wave Airflow Providers packages. This email

Re: [VOTE] Airflow Providers prepared on September 12, 2023

2023-09-12 Thread Jarek Potiuk
+1 (binding) - checked signatures, checksums, licences, sources On Tue, Sep 12, 2023 at 12:19 PM Elad Kalif wrote: > Hey all, > > I have just cut the *RC2* for dbt.cloud provider. This email is calling a > vote on the release, > which will last for 24 hours - which means that it will end on

Re: asking permission to edit airflow confluence page

2023-09-12 Thread Jarek Potiuk
Added you. Just be aware that you should likely subscribe to the devlist and discuss initial vision of what you want to propose - and before investing a lot of time into detailing it in the AIP, you can drop a question/start discussion here or maybe open a Github Discussion first (and obviously

Re: [VOTE] Airflow Providers prepared on September 08, 2023

2023-09-10 Thread Jarek Potiuk
+1 (binding) - checked my changes, signatures, licences, checksums, verified sources are the same in packages as in tags. On Mon, Sep 11, 2023 at 5:35 AM Phani Kumar wrote: > +1 non binding > > On Mon, 11 Sept 2023, 01:39 Pankaj Koti, .invalid> > wrote: > > > +1 (non-binding) > > > > 1. Tested

Re: [DISCUSS] Executors docs should be published in Airflow core or providers?

2023-09-08 Thread Jarek Potiuk
+1 On Fri, Sep 8, 2023 at 6:18 PM Daniel Standish wrote: > Sounds reasonable. >

Re: [LAZY CONSENSUS] - Removing the Experimental tag for Pluggy

2023-09-08 Thread Jarek Potiuk
to my > decision. > > +1 non binding from me. > > Thanks, > Amogh Desai > > On Fri, Sep 8, 2023, 13:49 Jarek Potiuk wrote: > > > One small thing: just make sure to use `[LAZY CONSENSUS]` in the subject > (I > > just changed it) > > > > Also I sugg

[LAZY CONSENSUS] - Removing the Experimental tag for Pluggy

2023-09-08 Thread Jarek Potiuk
One small thing: just make sure to use `[LAZY CONSENSUS]` in the subject (I just changed it) Also I suggest you subscribe to the devils with the email you have send it with - then you will receive responses (if there will be any) without the need of adding you back and we will not have to

<    1   2   3   4   5   6   7   8   9   10   >