Re: [DISCUSSION] Add 5 new Providers to enable first-class LLMOps

+1 from me I think the dashboard idea is great! On Thu, Oct 19, 2023 at 7:05 PM Andrey Anshin wrote: > Because 4 out 5 new providers have a draft PR I would like to raise a > question about which related to all new providers. Just to avoid the same > question in all PRs. > > Do we actually want

Re: [VOTE] AIP-58 Airflow ObjectStore

I like where this is heading, so I vote *+1*. Although, I would like to see some examples of usage in DAGs (before/after would be great) that will help support the following points that you have mentioned in the AIP

Re: [VOTE] AIP-58 Airflow ObjectStore

I dont mind waiting for that given a reasonable timeframe. Martin mentioned he wanted to do something at the end of the week. The vote to this AIP runs until next Thursday anyway :-). And thank you :-). B. On Thu, 19 Oct 2023 at 21:11, Jarek Potiuk wrote: > > One less worry I hope is that aiob

Re: [VOTE] AIP-58 Airflow ObjectStore

> One less worry I hope is that aiobotocore is actually starting to relax its botocore requirements bringing it much closer to latest release: https://github.com/aio-libs/aiobotocore/pull/1037 Oh yes absolutely. Great timing. And our constraints ***JUST*** caught up automatically with aiobotocore

Re: [VOTE] AIP-58 Airflow ObjectStore

Thanks for thorough consideration Jarek. I follow your concerns. The idea behind this AIP was to reduce the cognitive load on users by staying as much pythonic as we can and to be gentle with the Airflow-isms. So I hope to limit that "yet another abstraction". I do agree that having great examples

Re: [VOTE] AIP-58 Airflow ObjectStore

Finally caught up with this one, looked through code and discussions. I am a little torn on that one but I did some more research and I think it's a useful abstraction. +1(binding) The big + of using fsspec is that it is already supported by the most important "consumers" that are likely to be us

Re: [DISCUSSION] Add 5 new Providers to enable first-class LLMOps

Because 4 out 5 new providers have a draft PR I would like to raise a question about which related to all new providers. Just to avoid the same question in all PRs. Do we actually want to make new operators kindish of like "PythonOperator"? Maybe I miss some important thing and can't see why it w

Re: The "no_status" state

Like what Jarek said, some of these dependencies might take a lot of work to surface correctly. But I am happy to improve the grid and graph to show more information, like integrating rendered_templates and more details into the Grid view. Mind to open a github issue for some of those smaller tasks

Re: The "no_status" state

I think it will be tricky to get all the reasons surfaced to the user why the task is not run. But surfacing it to the user is indeed a good idea. Currently this is only done by this FAQ response - showing possible reasons https://airflow.apache.org/docs/apache-airflow/stable/faq.html#why-is-task-n

Re: [VOTE] AIP-58 Airflow ObjectStore

Thanks for incorporating the feedback! +1 (non-binding) On Thu, Oct 19, 2023 at 1:55 PM Dennis Akpenyi wrote: > +1 (non-binding) > > On Thu, Oct 19, 2023 at 12:24 PM Bolke de Bruin wrote: > > > Dear Community, > > > > I would like to start a vote for "AIP-58 Add Airflow ObjectStore". > > > > Y

Re: Airflow Docs Development Issues

Let me just clarify (because that could be unclear) what my +1 was about. I was not talking (and I believe Ryan was not talking either) about removing the old docs but about archiving them and serving from elsewhere (cloud storage). I think discussing changing to more shared HTML/JS/CSS is also a

Re: Airflow Docs Development Issues

Hey everyone, Thanks, Ryan for stating the thread :) Big +1 For archiving docs older than 18 months. We can still make the older docs available in `rst` doc form. But eventually, we might again run into this problem because of the growing no. of providers. I think the main reason for this issue

Re: Airflow Docs Development Issues

I am not happy about removing "old" docs. There are still users on older versions but given the situation I am not sure what other option we have. Maybe we should cut from a specific provider rather than from all of them? Why does Google provider consume 4 GB and Amazon 1.7 GB? Is there a specific

Re: [VOTE] AIP-58 Airflow ObjectStore

+1 (non-binding) On Thu, Oct 19, 2023 at 12:24 PM Bolke de Bruin wrote: > Dear Community, > > I would like to start a vote for "AIP-58 Add Airflow ObjectStore". > > You can find the AIP here: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263430565 > > Implementing PR (most o

Re: [DISCUSSION] Add 5 new Providers to enable first-class LLMOps

Note: Weaviate is also an open-source project with over 470k+ downloads last month . On Thu, 19 Oct 2023 at 12:40, Kaxil Naik wrote: > Absolutely, we will publish the results of test runs somewhere, we would >

Re: [DISCUSSION] Add 5 new Providers to enable first-class LLMOps

Absolutely, we will publish the results of test runs somewhere, we would probably start with dumping them in a publicly-accessible S3 bucket / Github issue and then move to a Dashboard. > Re 2) I think this is a great opportunity for Astronomers to take the > "3rd-party maintenance" role to follow

Re: [VOTE] AIP-58 Airflow ObjectStore

Going to look at it in detail today, Bolke :) On Thu, 19 Oct 2023 at 12:15, Avi wrote: > +1 (non-binding) > > Regards, > Avi > > > On Thursday, Oct 19, 2023 at 10:25, Bolke de Bruin (mailto:bdbr...@gmail.com)> wrote: > > Dear Community, > > > > I would like to start a vote for "AIP-58 Add Airfl

Re: [VOTE] AIP-58 Airflow ObjectStore

+1 (non-binding) Regards, Avi > On Thursday, Oct 19, 2023 at 10:25, Bolke de Bruin (mailto:bdbr...@gmail.com)> wrote: > Dear Community, > > I would like to start a vote for "AIP-58 Add Airflow ObjectStore". > > You can find the AIP here: > https://cwiki.apache.org/confluence/pages/viewpage.actio

Re: Airflow Docs Development Issues

Yes. Moving the old version to somewhere that we can keep/archive static historical versions of those historical docs and publish them from there. What you proposed is exactly the solution I thought might be best as well. It would be a great task to contribute to the stability of our docs generati

[VOTE] AIP-58 Airflow ObjectStore

Dear Community, I would like to start a vote for "AIP-58 Add Airflow ObjectStore". You can find the AIP here: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263430565 Implementing PR (most of the discussion happened here): https://github.com/apache/airflow/pull/34729 Discussio

Re: [VOTE] Airflow Providers prepared on October 18, 2023

+1 non-binding On Thu, Oct 19, 2023 at 1:25 PM Wei Lee wrote: > +1 non-binding > > Tested apache-airflow-providers-amazon==8.9.0rc1 and > apache-airflow-providers-google==10.10.1rc1. Both of them work as expected. > > Best, > Wei > > > On Oct 19, 2023, at 2:24 PM, Amogh Desai > wrote: > > > > +

Re: [VOTE] Airflow Providers prepared on October 18, 2023

+1 non-binding Tested apache-airflow-providers-amazon==8.9.0rc1 and apache-airflow-providers-google==10.10.1rc1. Both of them work as expected. Best, Wei > On Oct 19, 2023, at 2:24 PM, Amogh Desai wrote: > > +1 non binding > > Tested few test DAGs with cncf mostly. > > Thanks, > Amogh > >