Re: Unified Python Support

[Michael A. Smith]

All of that makes sense. I'd like to toss in one more suggestion: let's
release the python packages as wheels instead of tarballs or eggs.

https://pythonwheels.com has a good explanation for why.

I think there's already a ticket for this, although I can't go search for
it now. I'll change the build scripts in lang/py* sometime this week to do
that and to disable lang/py3 as well if there are no objections.

On Tue, Jan 7, 2020 at 02:40 Driesprong, Fokko  wrote:

> Makes sense Michael.
>
> I'm still working on this one:
> https://issues.apache.org/jira/browse/AVRO-2663
>
> The current fix that I came up with does not fix the root cause, but this
> is very complex. I'd like to fix this, and then start the release for
> 1.9.2. Hopefully, I have some time this weekend. Would that work for you?
>
> One more thing, for Airflow we had to release the package name from airflow
>  to apache-airflow
> . We've dropped the old one by
> throwing an Error when you try to import the package. Would this be
> something that we would like to do for Avro in the future? For example,
> releasing avro-python3 version 1.10.0 with the sole message of having to
> import the avro package? Would like to get your opinion on this.
>
> Cheers, Fokko
>
> Op ma 6 jan. 2020 om 13:32 schreef Michael A. Smith  >:
>
> > I'd suggest that we do at least one release that has support for both
> > python 2 and 3 in the same codebase. This may open doors for folks trying
> > to transition from both avro-python3 (lang/py3) to avro (lang/py) as well
> > as those trying to go from python 2 to 3 with lang/py.
> >
> > After that we should officially close out support for python 2.
> >
> > Please let me know how I can help with the release process. Should we
> have
> > a release soon?
> >
> > On Sun, Jan 5, 2020 at 13:31 Driesprong, Fokko 
> > wrote:
> >
> > > Thanks for bringing this up Michael and an awesome job on the Python
> > part.
> > >
> > > I'd suggest stopping releasing the avro-python3, and continue only
> > > releasing the avro package itself: https://pypi.org/project/avro/
> > >
> > > This will stop the releases of avro-python3, and in time we can also
> > remove
> > > it from the git repository. The big question is, are we still going to
> > > support Python2 for a while, it is still part of the CI. Supporting
> only
> > > higher versions of Python, such as 3.6, allows us to use new features,
> > such
> > > as type annotations.
> > >
> > > Cheers, Fokko
> > >
> > > Op zo 5 jan. 2020 om 18:44 schreef Michael A. Smith <
> > mich...@smith-li.com
> > > >:
> > >
> > > > Hi! Given that Python has ended support for python 2 as of the
> first, I
> > > > went ahead and merged the PR. Test coverage is pretty good, so I'm
> > fairly
> > > > confident; however this is a big change, involving nearly every
> module
> > in
> > > > the python part of the project.
> > > >
> > > > So I'm wondering how this works when it comes to releasing. There
> > aren't
> > > > any API changes in the literal implementation. So in that light there
> > > isn't
> > > > any need to treat this version specially. But Python itself is
> markedly
> > > > different between 2 and 3 in some relevant areas.
> > > >
> > > > Do we need to do anything different for the next release of the
> lang/py
> > > > codebase?
> > > >
> > > > Thanks for your guidance!
> > > >
> > > > On Fri, Dec 20, 2019 at 11:43 Ryan Skraba  wrote:
> > > >
> > > > > Hello!  I wanted to make sure to thank you for doing all this
> > > > > python2/3 work!  I've learned a lot by watching and reading the
> > Python
> > > > > PRs coming through.
> > > > >
> > > > > I did a rough pass through the types of changes and cleanup, and
> I'm
> > > > > pretty happy :D  I'll try to get more thorough pass done, but
> > (indeed)
> > > > > I probably won't have a lot of time between now and the new year.
> > > > >
> > > > > All my best, Ryan
> > > > >
> > > > > On Fri, Dec 20, 2019 at 12:36 AM Michael A. Smith <
> > > mich...@smith-li.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi, I've finished building out a unified python approach in
> > lang/py.
> > > > > > It passes our full Yetus tests in cpython 2.7 and 3.5. I also
> > tested
> > > > > > it and passed locally in 3.6, 3.7 and 3.8 as well as pypy 7.2.0
> for
> > > > > > both 2.7 and 3.6.
> > > > > >
> > > > > > The pull request is here:
> https://github.com/apache/avro/pull/744
> > > > > >
> > > > > > I know many people are on holiday or unavailable in the near
> > future,
> > > > > > but I would really appreciate some eyes on this if you can find
> the
> > > > > > time. The tests give me some confidence, but the change was a
> > > > > > significant lift, as Python3 and Python2 handle bytes and unicode
> > > > > > strings in substantially different ways.
> > > > > >
> > > > > > This gives us a path forward to unifying our python support (I
> > mean,
> > > > > > dropping lang/py3 and focusing on 

Re: Unified Python Support

[Driesprong, Fokko]

Makes sense Michael.

I'm still working on this one:
https://issues.apache.org/jira/browse/AVRO-2663

The current fix that I came up with does not fix the root cause, but this
is very complex. I'd like to fix this, and then start the release for
1.9.2. Hopefully, I have some time this weekend. Would that work for you?

One more thing, for Airflow we had to release the package name from airflow
 to apache-airflow
. We've dropped the old one by
throwing an Error when you try to import the package. Would this be
something that we would like to do for Avro in the future? For example,
releasing avro-python3 version 1.10.0 with the sole message of having to
import the avro package? Would like to get your opinion on this.

Cheers, Fokko

Op ma 6 jan. 2020 om 13:32 schreef Michael A. Smith :

> I'd suggest that we do at least one release that has support for both
> python 2 and 3 in the same codebase. This may open doors for folks trying
> to transition from both avro-python3 (lang/py3) to avro (lang/py) as well
> as those trying to go from python 2 to 3 with lang/py.
>
> After that we should officially close out support for python 2.
>
> Please let me know how I can help with the release process. Should we have
> a release soon?
>
> On Sun, Jan 5, 2020 at 13:31 Driesprong, Fokko 
> wrote:
>
> > Thanks for bringing this up Michael and an awesome job on the Python
> part.
> >
> > I'd suggest stopping releasing the avro-python3, and continue only
> > releasing the avro package itself: https://pypi.org/project/avro/
> >
> > This will stop the releases of avro-python3, and in time we can also
> remove
> > it from the git repository. The big question is, are we still going to
> > support Python2 for a while, it is still part of the CI. Supporting only
> > higher versions of Python, such as 3.6, allows us to use new features,
> such
> > as type annotations.
> >
> > Cheers, Fokko
> >
> > Op zo 5 jan. 2020 om 18:44 schreef Michael A. Smith <
> mich...@smith-li.com
> > >:
> >
> > > Hi! Given that Python has ended support for python 2 as of the first, I
> > > went ahead and merged the PR. Test coverage is pretty good, so I'm
> fairly
> > > confident; however this is a big change, involving nearly every module
> in
> > > the python part of the project.
> > >
> > > So I'm wondering how this works when it comes to releasing. There
> aren't
> > > any API changes in the literal implementation. So in that light there
> > isn't
> > > any need to treat this version specially. But Python itself is markedly
> > > different between 2 and 3 in some relevant areas.
> > >
> > > Do we need to do anything different for the next release of the lang/py
> > > codebase?
> > >
> > > Thanks for your guidance!
> > >
> > > On Fri, Dec 20, 2019 at 11:43 Ryan Skraba  wrote:
> > >
> > > > Hello!  I wanted to make sure to thank you for doing all this
> > > > python2/3 work!  I've learned a lot by watching and reading the
> Python
> > > > PRs coming through.
> > > >
> > > > I did a rough pass through the types of changes and cleanup, and I'm
> > > > pretty happy :D  I'll try to get more thorough pass done, but
> (indeed)
> > > > I probably won't have a lot of time between now and the new year.
> > > >
> > > > All my best, Ryan
> > > >
> > > > On Fri, Dec 20, 2019 at 12:36 AM Michael A. Smith <
> > mich...@smith-li.com>
> > > > wrote:
> > > > >
> > > > > Hi, I've finished building out a unified python approach in
> lang/py.
> > > > > It passes our full Yetus tests in cpython 2.7 and 3.5. I also
> tested
> > > > > it and passed locally in 3.6, 3.7 and 3.8 as well as pypy 7.2.0 for
> > > > > both 2.7 and 3.6.
> > > > >
> > > > > The pull request is here: https://github.com/apache/avro/pull/744
> > > > >
> > > > > I know many people are on holiday or unavailable in the near
> future,
> > > > > but I would really appreciate some eyes on this if you can find the
> > > > > time. The tests give me some confidence, but the change was a
> > > > > significant lift, as Python3 and Python2 handle bytes and unicode
> > > > > strings in substantially different ways.
> > > > >
> > > > > This gives us a path forward to unifying our python support (I
> mean,
> > > > > dropping lang/py3 and focusing on one API in one place) as well as
> > > > > managing the sunset of python 2 support altogether.
> > > > >
> > > > > Thank you for your help with this project, either way!
> > > > >
> > > > > - Michael
> > > >
> > >
> >
>

Re: Unified Python Support

[Sean Busbey]

Yeah we should have a release; we are overdue.

On Mon, Jan 6, 2020, 06:32 Michael A. Smith  wrote:

> I'd suggest that we do at least one release that has support for both
> python 2 and 3 in the same codebase. This may open doors for folks trying
> to transition from both avro-python3 (lang/py3) to avro (lang/py) as well
> as those trying to go from python 2 to 3 with lang/py.
>
> After that we should officially close out support for python 2.
>
> Please let me know how I can help with the release process. Should we have
> a release soon?
>
> On Sun, Jan 5, 2020 at 13:31 Driesprong, Fokko 
> wrote:
>
> > Thanks for bringing this up Michael and an awesome job on the Python
> part.
> >
> > I'd suggest stopping releasing the avro-python3, and continue only
> > releasing the avro package itself: https://pypi.org/project/avro/
> >
> > This will stop the releases of avro-python3, and in time we can also
> remove
> > it from the git repository. The big question is, are we still going to
> > support Python2 for a while, it is still part of the CI. Supporting only
> > higher versions of Python, such as 3.6, allows us to use new features,
> such
> > as type annotations.
> >
> > Cheers, Fokko
> >
> > Op zo 5 jan. 2020 om 18:44 schreef Michael A. Smith <
> mich...@smith-li.com
> > >:
> >
> > > Hi! Given that Python has ended support for python 2 as of the first, I
> > > went ahead and merged the PR. Test coverage is pretty good, so I'm
> fairly
> > > confident; however this is a big change, involving nearly every module
> in
> > > the python part of the project.
> > >
> > > So I'm wondering how this works when it comes to releasing. There
> aren't
> > > any API changes in the literal implementation. So in that light there
> > isn't
> > > any need to treat this version specially. But Python itself is markedly
> > > different between 2 and 3 in some relevant areas.
> > >
> > > Do we need to do anything different for the next release of the lang/py
> > > codebase?
> > >
> > > Thanks for your guidance!
> > >
> > > On Fri, Dec 20, 2019 at 11:43 Ryan Skraba  wrote:
> > >
> > > > Hello!  I wanted to make sure to thank you for doing all this
> > > > python2/3 work!  I've learned a lot by watching and reading the
> Python
> > > > PRs coming through.
> > > >
> > > > I did a rough pass through the types of changes and cleanup, and I'm
> > > > pretty happy :D  I'll try to get more thorough pass done, but
> (indeed)
> > > > I probably won't have a lot of time between now and the new year.
> > > >
> > > > All my best, Ryan
> > > >
> > > > On Fri, Dec 20, 2019 at 12:36 AM Michael A. Smith <
> > mich...@smith-li.com>
> > > > wrote:
> > > > >
> > > > > Hi, I've finished building out a unified python approach in
> lang/py.
> > > > > It passes our full Yetus tests in cpython 2.7 and 3.5. I also
> tested
> > > > > it and passed locally in 3.6, 3.7 and 3.8 as well as pypy 7.2.0 for
> > > > > both 2.7 and 3.6.
> > > > >
> > > > > The pull request is here: https://github.com/apache/avro/pull/744
> > > > >
> > > > > I know many people are on holiday or unavailable in the near
> future,
> > > > > but I would really appreciate some eyes on this if you can find the
> > > > > time. The tests give me some confidence, but the change was a
> > > > > significant lift, as Python3 and Python2 handle bytes and unicode
> > > > > strings in substantially different ways.
> > > > >
> > > > > This gives us a path forward to unifying our python support (I
> mean,
> > > > > dropping lang/py3 and focusing on one API in one place) as well as
> > > > > managing the sunset of python 2 support altogether.
> > > > >
> > > > > Thank you for your help with this project, either way!
> > > > >
> > > > > - Michael
> > > >
> > >
> >
>

Re: Unified Python Support

[Michael A. Smith]

I'd suggest that we do at least one release that has support for both
python 2 and 3 in the same codebase. This may open doors for folks trying
to transition from both avro-python3 (lang/py3) to avro (lang/py) as well
as those trying to go from python 2 to 3 with lang/py.

After that we should officially close out support for python 2.

Please let me know how I can help with the release process. Should we have
a release soon?

On Sun, Jan 5, 2020 at 13:31 Driesprong, Fokko  wrote:

> Thanks for bringing this up Michael and an awesome job on the Python part.
>
> I'd suggest stopping releasing the avro-python3, and continue only
> releasing the avro package itself: https://pypi.org/project/avro/
>
> This will stop the releases of avro-python3, and in time we can also remove
> it from the git repository. The big question is, are we still going to
> support Python2 for a while, it is still part of the CI. Supporting only
> higher versions of Python, such as 3.6, allows us to use new features, such
> as type annotations.
>
> Cheers, Fokko
>
> Op zo 5 jan. 2020 om 18:44 schreef Michael A. Smith  >:
>
> > Hi! Given that Python has ended support for python 2 as of the first, I
> > went ahead and merged the PR. Test coverage is pretty good, so I'm fairly
> > confident; however this is a big change, involving nearly every module in
> > the python part of the project.
> >
> > So I'm wondering how this works when it comes to releasing. There aren't
> > any API changes in the literal implementation. So in that light there
> isn't
> > any need to treat this version specially. But Python itself is markedly
> > different between 2 and 3 in some relevant areas.
> >
> > Do we need to do anything different for the next release of the lang/py
> > codebase?
> >
> > Thanks for your guidance!
> >
> > On Fri, Dec 20, 2019 at 11:43 Ryan Skraba  wrote:
> >
> > > Hello!  I wanted to make sure to thank you for doing all this
> > > python2/3 work!  I've learned a lot by watching and reading the Python
> > > PRs coming through.
> > >
> > > I did a rough pass through the types of changes and cleanup, and I'm
> > > pretty happy :D  I'll try to get more thorough pass done, but (indeed)
> > > I probably won't have a lot of time between now and the new year.
> > >
> > > All my best, Ryan
> > >
> > > On Fri, Dec 20, 2019 at 12:36 AM Michael A. Smith <
> mich...@smith-li.com>
> > > wrote:
> > > >
> > > > Hi, I've finished building out a unified python approach in lang/py.
> > > > It passes our full Yetus tests in cpython 2.7 and 3.5. I also tested
> > > > it and passed locally in 3.6, 3.7 and 3.8 as well as pypy 7.2.0 for
> > > > both 2.7 and 3.6.
> > > >
> > > > The pull request is here: https://github.com/apache/avro/pull/744
> > > >
> > > > I know many people are on holiday or unavailable in the near future,
> > > > but I would really appreciate some eyes on this if you can find the
> > > > time. The tests give me some confidence, but the change was a
> > > > significant lift, as Python3 and Python2 handle bytes and unicode
> > > > strings in substantially different ways.
> > > >
> > > > This gives us a path forward to unifying our python support (I mean,
> > > > dropping lang/py3 and focusing on one API in one place) as well as
> > > > managing the sunset of python 2 support altogether.
> > > >
> > > > Thank you for your help with this project, either way!
> > > >
> > > > - Michael
> > >
> >
>

Re: Unified Python Support

[Driesprong, Fokko]

Thanks for bringing this up Michael and an awesome job on the Python part.

I'd suggest stopping releasing the avro-python3, and continue only
releasing the avro package itself: https://pypi.org/project/avro/

This will stop the releases of avro-python3, and in time we can also remove
it from the git repository. The big question is, are we still going to
support Python2 for a while, it is still part of the CI. Supporting only
higher versions of Python, such as 3.6, allows us to use new features, such
as type annotations.

Cheers, Fokko

Op zo 5 jan. 2020 om 18:44 schreef Michael A. Smith :

> Hi! Given that Python has ended support for python 2 as of the first, I
> went ahead and merged the PR. Test coverage is pretty good, so I'm fairly
> confident; however this is a big change, involving nearly every module in
> the python part of the project.
>
> So I'm wondering how this works when it comes to releasing. There aren't
> any API changes in the literal implementation. So in that light there isn't
> any need to treat this version specially. But Python itself is markedly
> different between 2 and 3 in some relevant areas.
>
> Do we need to do anything different for the next release of the lang/py
> codebase?
>
> Thanks for your guidance!
>
> On Fri, Dec 20, 2019 at 11:43 Ryan Skraba  wrote:
>
> > Hello!  I wanted to make sure to thank you for doing all this
> > python2/3 work!  I've learned a lot by watching and reading the Python
> > PRs coming through.
> >
> > I did a rough pass through the types of changes and cleanup, and I'm
> > pretty happy :D  I'll try to get more thorough pass done, but (indeed)
> > I probably won't have a lot of time between now and the new year.
> >
> > All my best, Ryan
> >
> > On Fri, Dec 20, 2019 at 12:36 AM Michael A. Smith 
> > wrote:
> > >
> > > Hi, I've finished building out a unified python approach in lang/py.
> > > It passes our full Yetus tests in cpython 2.7 and 3.5. I also tested
> > > it and passed locally in 3.6, 3.7 and 3.8 as well as pypy 7.2.0 for
> > > both 2.7 and 3.6.
> > >
> > > The pull request is here: https://github.com/apache/avro/pull/744
> > >
> > > I know many people are on holiday or unavailable in the near future,
> > > but I would really appreciate some eyes on this if you can find the
> > > time. The tests give me some confidence, but the change was a
> > > significant lift, as Python3 and Python2 handle bytes and unicode
> > > strings in substantially different ways.
> > >
> > > This gives us a path forward to unifying our python support (I mean,
> > > dropping lang/py3 and focusing on one API in one place) as well as
> > > managing the sunset of python 2 support altogether.
> > >
> > > Thank you for your help with this project, either way!
> > >
> > > - Michael
> >
>

Re: Unified Python Support

[Michael A. Smith]

Hi! Given that Python has ended support for python 2 as of the first, I
went ahead and merged the PR. Test coverage is pretty good, so I'm fairly
confident; however this is a big change, involving nearly every module in
the python part of the project.

So I'm wondering how this works when it comes to releasing. There aren't
any API changes in the literal implementation. So in that light there isn't
any need to treat this version specially. But Python itself is markedly
different between 2 and 3 in some relevant areas.

Do we need to do anything different for the next release of the lang/py
codebase?

Thanks for your guidance!

On Fri, Dec 20, 2019 at 11:43 Ryan Skraba  wrote:

> Hello!  I wanted to make sure to thank you for doing all this
> python2/3 work!  I've learned a lot by watching and reading the Python
> PRs coming through.
>
> I did a rough pass through the types of changes and cleanup, and I'm
> pretty happy :D  I'll try to get more thorough pass done, but (indeed)
> I probably won't have a lot of time between now and the new year.
>
> All my best, Ryan
>
> On Fri, Dec 20, 2019 at 12:36 AM Michael A. Smith 
> wrote:
> >
> > Hi, I've finished building out a unified python approach in lang/py.
> > It passes our full Yetus tests in cpython 2.7 and 3.5. I also tested
> > it and passed locally in 3.6, 3.7 and 3.8 as well as pypy 7.2.0 for
> > both 2.7 and 3.6.
> >
> > The pull request is here: https://github.com/apache/avro/pull/744
> >
> > I know many people are on holiday or unavailable in the near future,
> > but I would really appreciate some eyes on this if you can find the
> > time. The tests give me some confidence, but the change was a
> > significant lift, as Python3 and Python2 handle bytes and unicode
> > strings in substantially different ways.
> >
> > This gives us a path forward to unifying our python support (I mean,
> > dropping lang/py3 and focusing on one API in one place) as well as
> > managing the sunset of python 2 support altogether.
> >
> > Thank you for your help with this project, either way!
> >
> > - Michael
>

Re: Unified Python Support

[Ryan Skraba]

Hello!  I wanted to make sure to thank you for doing all this
python2/3 work!  I've learned a lot by watching and reading the Python
PRs coming through.

I did a rough pass through the types of changes and cleanup, and I'm
pretty happy :D  I'll try to get more thorough pass done, but (indeed)
I probably won't have a lot of time between now and the new year.

All my best, Ryan

On Fri, Dec 20, 2019 at 12:36 AM Michael A. Smith  wrote:
>
> Hi, I've finished building out a unified python approach in lang/py.
> It passes our full Yetus tests in cpython 2.7 and 3.5. I also tested
> it and passed locally in 3.6, 3.7 and 3.8 as well as pypy 7.2.0 for
> both 2.7 and 3.6.
>
> The pull request is here: https://github.com/apache/avro/pull/744
>
> I know many people are on holiday or unavailable in the near future,
> but I would really appreciate some eyes on this if you can find the
> time. The tests give me some confidence, but the change was a
> significant lift, as Python3 and Python2 handle bytes and unicode
> strings in substantially different ways.
>
> This gives us a path forward to unifying our python support (I mean,
> dropping lang/py3 and focusing on one API in one place) as well as
> managing the sunset of python 2 support altogether.
>
> Thank you for your help with this project, either way!
>
> - Michael

Unified Python Support

[Michael A. Smith]

Hi, I've finished building out a unified python approach in lang/py.
It passes our full Yetus tests in cpython 2.7 and 3.5. I also tested
it and passed locally in 3.6, 3.7 and 3.8 as well as pypy 7.2.0 for
both 2.7 and 3.6.

The pull request is here: https://github.com/apache/avro/pull/744

I know many people are on holiday or unavailable in the near future,
but I would really appreciate some eyes on this if you can find the
time. The tests give me some confidence, but the change was a
significant lift, as Python3 and Python2 handle bytes and unicode
strings in substantially different ways.

This gives us a path forward to unifying our python support (I mean,
dropping lang/py3 and focusing on one API in one place) as well as
managing the sunset of python 2 support altogether.

Thank you for your help with this project, either way!

- Michael