This vote is cancelled. RC3 will follow On Wed, Apr 26, 2023 at 4:23 AM Mehta, Shubham <[email protected]> wrote:
> Thank Jarek, good catch! You're right, it's better to keep aiobotocore as > additional dependency for now. > If you attempt to install apache-airflow-providers-amazon[aiobotocore] > using constraints for older Airflow versions, like Pankaj, you will still > encounter conflicts. However, the impact would be less severe, as the use > of Amazon's deferrable functionality would be minimal in the beginning. > > > On 2023-04-25, 5:15 PM, "Jarek Potiuk" <[email protected] <mailto: > [email protected]>> wrote: > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > PR here: https://github.com/apache/airflow/pull/30874 < > https://github.com/apache/airflow/pull/30874> > > > On Wed, Apr 26, 2023 at 2:08 AM Jarek Potiuk <[email protected] <mailto: > [email protected]>> wrote: > > > > -1: Pankaj is right. We need to fix the dependencies. The aiobotocore was > > mistakenly added as the "required" dependency 2 days ago in: > > https://github.com/apache/airflow/pull/30032 < > https://github.com/apache/airflow/pull/30032> > > > > However for the Amazon provider we have it as an optional one. > > > > Currently in provider,yaml we have both: > > > > dependencies: > > .... > > - aiobotocore[boto3]>=2.2.0 > > > > > > and: > > > > additional-extras: > > - name: pandas > > dependencies: > > - pandas>=0.17.1 > > # There is conflict between boto3 and aiobotocore dependency botocore. > > # TODO: We can remove it once boto3 and aiobotocore both have compatible > botocore version or > > # boto3 have native async support and we move away from aio aiobotocore > > - name: aiobotocore > > dependencies: > > - aiobotocore>=2.1.1 > > > > > > The first one should be removed and the second one updated. PR is coming. > > > > J. > > > > > > > > > > > > On Wed, Apr 26, 2023 at 12:26 AM Mehta, Shubham <[email protected] > <mailto:[email protected]>lid> > > wrote: > > > >> Hey Pankaj, > >> > >> I have a few clarifying questions: > >> 1. Can you specify the Airflow version used during testing? Based on the > >> botocore version constraint, it appears to be v2.5.3. > >> 2. Did you use a modified constraint file to install the Amazon provider > >> v8.0.0rc2 with Airflow v2.5.3? If so, what were the modifications? > >> > >> As you know, we added the _aiobotocore_ dependency to support deferrable > >> operators in the Amazon provider package (AMPP) because boto3 does not > >> offer async support by default. This decision has some implications, > which > >> we discussed in detail within the related GitHub issues: > >> 1. _aiobotocore_ is included as a dependency in the constraints file > >> starting from Airflow v2.6.0 and onwards. Consequently, users will not > >> encounter version conflict issues for any Airflow version equal to or > >> greater than 2.6.0. > >> 2. Since _aiobotocore_ was not added as a dependency for older Airflow > >> versions, conflicts may arise if you attempt to use newer AMPP versions > by > >> selectively modifying the constraint file for the AMPP version alone, > >> without adjusting botocore and boto3 versions. > >> > >> In summary, for older Airflow versions, you must modify the constraints > >> for AMPP, botocore, and boto3. For newer Airflow versions, the process > will > >> remain smooth, as it was before. Although this may result in a slightly > >> worse user experience, it is the only alternative we have in order to > >> support deferrable operators in AMPP. > >> > >> Thanks > >> Shubham > >> > >> On 2023-04-25, 11:57 AM, "Pankaj Singh" <[email protected] > <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>>> > wrote: > >> > >> > >> CAUTION: This email originated from outside of the organization. Do not > >> click links or open attachments unless you can confirm the sender and > know > >> the content is safe. > >> > >> > >> > >> > >> > >> > >> -1 (non-binding) > >> > >> > >> We won't be able to use the amazon provider for airflow version less > than > >> equal to 2.5.3 because conflict cause by aibotocore > >> > >> > >> > >> > >> ERROR: Cannot install aiobotocore[boto3]==2.2.0, > >> aiobotocore[boto3]==2.3.0, > >> aiobotocore[boto3]==2.3.1, aiobotocore[boto3]==2.3.2, > >> aiobotocore[boto3]==2.3.3, aiobotocore[boto3]==2.3.4, > >> aiobotocore[boto3]==2.4.0, aiobotocore[boto3]==2.4.1, > >> aiobotocore[boto3]==2.4.2 and aiobotocore[boto3]==2.5.0 because these > >> package versions have conflicting dependencies. > >> > >> > >> The conflict is caused by: > >> aiobotocore[boto3] 2.5.0 depends on botocore<1.29.77 and >=1.29.76 > >> aiobotocore[boto3] 2.4.2 depends on botocore<1.27.60 and >=1.27.59 > >> aiobotocore[boto3] 2.4.1 depends on botocore<1.27.60 and >=1.27.59 > >> aiobotocore[boto3] 2.4.0 depends on botocore<1.27.60 and >=1.27.59 > >> aiobotocore[boto3] 2.3.4 depends on botocore<1.24.22 and >=1.24.21 > >> aiobotocore[boto3] 2.3.3 depends on botocore<1.24.22 and >=1.24.21 > >> aiobotocore[boto3] 2.3.2 depends on botocore<1.24.22 and >=1.24.21 > >> aiobotocore[boto3] 2.3.1 depends on botocore<1.24.22 and >=1.24.21 > >> aiobotocore[boto3] 2.3.0 depends on botocore<1.24.22 and >=1.24.21 > >> aiobotocore[boto3] 2.2.0 depends on botocore<1.24.22 and >=1.24.21 > >> The user requested (constraint) botocore==1.29.102 > >> > >> > >> On Tue, Apr 25, 2023 at 8:35 PM Jed Cunningham < > [email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>>> > >> wrote: > >> > >> > >> > +1 (binding) > >> > > >> > Checked signatures, checksums, licences. > >> > > >> > >> > >> > >> > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] >
