+1 (non-binding ) for Amazon, google, and azure verified our example dags
worked fine.
Regards,
Rahul Vats
9953794332
On Tue, 14 May 2024 at 09:45, Wei Lee wrote:
> +1 (non-binding)
>
> Test a few examples DAGs with Amazon, google, and azure providers without
> encountering issues.
>
> Best,
Definitely a fast moving thread on the mailing list. I haven’t been able to
respond for a few days and feel very far behind already.
A few comments on topics discussed the last few days:
- Jarek, in response to your comments around being more aggressive than in
Airflow 2 about deprecation and
Super-excited about that.
Question/Proposal: Can we have it possible to have two (or maybe three -
like a sub-committee) co-owners of topics? I think it's a lot to put on
one's head to "own" a topic and given circumstances/ volunteer time of
people, interruptions (and life intervening), it might
+1 (non-binding)
Test a few examples DAGs with Amazon, google, and azure providers without
encountering issues.
Best,
Wei
> On May 13, 2024, at 9:44 PM, Hussein Awala wrote:
>
> +1 (binding): checked the licences, the signatures, the checksums, and ran
> some testing dags for Amazon
Thank you all, I am very happy about the discussions.
The mailing list moves fast :). The main reason I recommended starting the
dev calls in early June was to have some of these discussions on the
mailing list.
Since Michal already scheduled a call, let's start there to discuss
various ideas.
Declaring connections prior to task execution was already proposed in AIP-1
:-). At that time, I had in mind to communicate over IPC to the task the
required settings. Registration could then happen with a manifest. Maybe
during DAG serialization this could be obtained unobtrusively? The benefit
> That would require some mechanism of declaring prior to task execution what
> connections would be used
That’s exactly what I’m proposing in the proposal doc I’m working on (It’s part
of also overhauling and re-designing the “Task Execution interface” that also
gives us the ability to nicely
re
As tasks require connection access, I assume connection data will somehow
> be passed as part of the
> metadata to task execution - whether it's part of the executor protocol or
> in some other way (I'm
> not an expert on that part of Airflow). Then, provided it's accessible as
> part of some
OK. Tests should be green now - all the issues are "handled" - there are
few follow up tasks from the tests run on 2.9.1 but the PR should be quite
ready for final review and merge now and I can attempt to look at 2.8
compatibility once it's done.
On Mon, May 13, 2024 at 1:17 AM Jarek Potiuk
+1 (binding): checked the licences, the signatures, the checksums, and ran
some testing dags for Amazon provider.
On Monday, May 13, 2024, Jarek Potiuk wrote:
> +1 (binding): verified reproducibility, signatures, checksums, licences.
>
> On Sun, May 12, 2024 at 9:34 PM Elad Kalif wrote:
>
> >
I think relying on connections wrapped in the task execution context makes
sense as a way to retrieve the connections to be used by OpenLineage.
Similarly, lineage backend relied on the task instance context in the past (
+1 (binding): verified reproducibility, signatures, checksums, licences.
On Sun, May 12, 2024 at 9:34 PM Elad Kalif wrote:
> Hey all,
>
> I have just cut the new wave Airflow Providers packages. This email is
> calling a vote on the release,
> which will last for 72 hours - which means that it
I would like to discuss the role of the Listener API and the OpenLineage
provider that utilizes it,
as there are current discussions that impact this integration.
For example, there is a consensus on (and I strongly support) removing
direct database access from
workers. However, currently,
Thanks to everyone for the constructive discussion - let's make it even
more specific in the weekly syncs! I put a slot for us next Tuesday. I had
some challenges with how some of the e-mails are displayed in this thread -
please reach out to me if you'd like to be added.
In the meantime, I
14 matches
Mail list logo
