Re: [ANNOUNCE] new committer Michael Smith

[Niels Basjes]

Welcome!

On Sun, Oct 20, 2019, 05:20 Sean Busbey  wrote:

> Hi folks!
>
> I'm very pleased to announce that Michael Smith has accepted the PMC's
> invitation to become a committer on the Apache Avro project.
>
> Please join me in congratulating Michael on this recognition of their
> great work thus far in our community.
>

[jira] [Comment Edited] (AVRO-1788) python api differs unnecessarily between python2 and python3

[Michael A. Smith (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955536#comment-16955536
 ] 

Michael A. Smith edited comment on AVRO-1788 at 10/20/19 4:25 PM:
--

I have made [an incremental PR#683|https://github.com/apache/avro/pull/683] to 
improve future compatibility of the Python 2 codebase. It doesn't entirely 
resolve the differences, but it resolves some of the low-hanging fruit in terms 
of being able to use modern static analysis.

It does remove support for Python versions < 2.7.

Avro is particularly sensitive to the bytes vs strings vs unicode differences 
between Python2 and 3, so that will take a fair bit of effort and is not 
included in this PR.


was (Author: kojiromike):
I have made [an incremental PR#683|https://github.com/apache/avro/pull/683] to 
improve future compatibility of the Python 2 codebase. It doesn't entirely 
resolve the differences, but it resolves some of the low-hanging fruit in terms 
of being able to use modern static analysis.

It does remove support for Python versions < 2.7.

Avro is particularly sensitive to the bytes <-> strings <-> unicode differences 
between Python2 and 3, so that will take a fair bit of effort and is not 
included in this PR.

> python api differs unnecessarily between python2 and python3
> 
>
> Key: AVRO-1788
> URL: https://issues.apache.org/jira/browse/AVRO-1788
> Project: Apache Avro
>  Issue Type: Wish
>  Components: python
>Reporter: Alexander Hasha
>Assignee: Michael A. Smith
>Priority: Minor
>
> It is difficult to use the python avro module in projects that maintain 
> simultaneous compatibility between python 2 and python 3, because there are 
> small differences in the API between the two versions in the module that seem 
> unnecessary.
> For example, to parse a schema string in python 2:
> {code}
> from avro.schema import parse
> {code}
> and in python 3:
> {code}
> from avro.schema import Parse
> {code}
> In Python2, DatumReader's constructor has keyword argument "writers_schema", 
> and in Python 3 it becomes "writer_schema" (no s).
> Would it be possible to align method names and arguments across the two 
> modules?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AVRO-1788) python api differs unnecessarily between python2 and python3

[Michael A. Smith (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955536#comment-16955536
 ] 

Michael A. Smith commented on AVRO-1788:


I have made [an incremental PR#683|https://github.com/apache/avro/pull/683] to 
improve future compatibility of the Python 2 codebase. It doesn't entirely 
resolve the differences, but it resolves some of the low-hanging fruit in terms 
of being able to use modern static analysis.

It does remove support for Python versions < 2.7.

Avro is particularly sensitive to the bytes <-> strings <-> unicode differences 
between Python2 and 3, so that will take a fair bit of effort and is not 
included in this PR.

> python api differs unnecessarily between python2 and python3
> 
>
> Key: AVRO-1788
> URL: https://issues.apache.org/jira/browse/AVRO-1788
> Project: Apache Avro
>  Issue Type: Wish
>  Components: python
>Reporter: Alexander Hasha
>Assignee: Michael A. Smith
>Priority: Minor
>
> It is difficult to use the python avro module in projects that maintain 
> simultaneous compatibility between python 2 and python 3, because there are 
> small differences in the API between the two versions in the module that seem 
> unnecessary.
> For example, to parse a schema string in python 2:
> {code}
> from avro.schema import parse
> {code}
> and in python 3:
> {code}
> from avro.schema import Parse
> {code}
> In Python2, DatumReader's constructor has keyword argument "writers_schema", 
> and in Python 3 it becomes "writer_schema" (no s).
> Would it be possible to align method names and arguments across the two 
> modules?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AVRO-1788) python api differs unnecessarily between python2 and python3

[Michael A. Smith (Jira)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael A. Smith updated AVRO-1788:
---
Status: In Progress  (was: Patch Available)

> python api differs unnecessarily between python2 and python3
> 
>
> Key: AVRO-1788
> URL: https://issues.apache.org/jira/browse/AVRO-1788
> Project: Apache Avro
>  Issue Type: Wish
>  Components: python
>Reporter: Alexander Hasha
>Assignee: Michael A. Smith
>Priority: Minor
>
> It is difficult to use the python avro module in projects that maintain 
> simultaneous compatibility between python 2 and python 3, because there are 
> small differences in the API between the two versions in the module that seem 
> unnecessary.
> For example, to parse a schema string in python 2:
> {code}
> from avro.schema import parse
> {code}
> and in python 3:
> {code}
> from avro.schema import Parse
> {code}
> In Python2, DatumReader's constructor has keyword argument "writers_schema", 
> and in Python 3 it becomes "writer_schema" (no s).
> Would it be possible to align method names and arguments across the two 
> modules?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (AVRO-1788) python api differs unnecessarily between python2 and python3

[Michael A. Smith (Jira)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael A. Smith reassigned AVRO-1788:
--

Assignee: Michael A. Smith

> python api differs unnecessarily between python2 and python3
> 
>
> Key: AVRO-1788
> URL: https://issues.apache.org/jira/browse/AVRO-1788
> Project: Apache Avro
>  Issue Type: Wish
>  Components: python
>Reporter: Alexander Hasha
>Assignee: Michael A. Smith
>Priority: Minor
>
> It is difficult to use the python avro module in projects that maintain 
> simultaneous compatibility between python 2 and python 3, because there are 
> small differences in the API between the two versions in the module that seem 
> unnecessary.
> For example, to parse a schema string in python 2:
> {code}
> from avro.schema import parse
> {code}
> and in python 3:
> {code}
> from avro.schema import Parse
> {code}
> In Python2, DatumReader's constructor has keyword argument "writers_schema", 
> and in Python 3 it becomes "writer_schema" (no s).
> Would it be possible to align method names and arguments across the two 
> modules?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (AVRO-2601) Add external codecs to extras_require

[Michael A. Smith (Jira)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael A. Smith reassigned AVRO-2601:
--

Assignee: Michael A. Smith

> Add external codecs to extras_require
> -
>
> Key: AVRO-2601
> URL: https://issues.apache.org/jira/browse/AVRO-2601
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: python
>Reporter: Michael A. Smith
>Assignee: Michael A. Smith
>Priority: Minor
>
> The Python implementations support some compression algorithms that aren't 
> supported in the python standard library. Users would need to install these 
> codecs themselves to use them. To make that process easier, we can add these 
> optional codecs to extras_require in the setup scripts so that users can 
> require `avro[zstandard]` instead of separately installing `avro` and 
> `zstandard`.
>  
> There's precedent here, since this is how avro already does it for snappy. 
> Also, encouraging installing zstandard or snappy via extras_require can 
> reduce the risk of compatibility issues.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work started] (AVRO-2601) Add external codecs to extras_require

[Michael A. Smith (Jira)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AVRO-2601 started by Michael A. Smith.
--
> Add external codecs to extras_require
> -
>
> Key: AVRO-2601
> URL: https://issues.apache.org/jira/browse/AVRO-2601
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: python
>Reporter: Michael A. Smith
>Assignee: Michael A. Smith
>Priority: Minor
>
> The Python implementations support some compression algorithms that aren't 
> supported in the python standard library. Users would need to install these 
> codecs themselves to use them. To make that process easier, we can add these 
> optional codecs to extras_require in the setup scripts so that users can 
> require `avro[zstandard]` instead of separately installing `avro` and 
> `zstandard`.
>  
> There's precedent here, since this is how avro already does it for snappy. 
> Also, encouraging installing zstandard or snappy via extras_require can 
> reduce the risk of compatibility issues.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AVRO-2601) Add external codecs to extras_require

[Michael A. Smith (Jira)]

Michael A. Smith created AVRO-2601:
--

 Summary: Add external codecs to extras_require
 Key: AVRO-2601
 URL: https://issues.apache.org/jira/browse/AVRO-2601
 Project: Apache Avro
  Issue Type: Improvement
  Components: python
Reporter: Michael A. Smith


The Python implementations support some compression algorithms that aren't 
supported in the python standard library. Users would need to install these 
codecs themselves to use them. To make that process easier, we can add these 
optional codecs to extras_require in the setup scripts so that users can 
require `avro[zstandard]` instead of separately installing `avro` and 
`zstandard`.

 

There's precedent here, since this is how avro already does it for snappy. 
Also, encouraging installing zstandard or snappy via extras_require can reduce 
the risk of compatibility issues.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AVRO-2591) Getting error when reading avro message python

[Michael A. Smith (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955519#comment-16955519
 ] 

Michael A. Smith commented on AVRO-2591:


Tried to make a comment and Jira apparently thought I was typing hotkeys. Sorry 
for the assign/unassign spam. Anyway, I was going to say that this problem 
looks suspiciously like what I see when I try to use BytesIO instead of 
StringIO in the Python 2 code in order to make that codebase more future 
compatible. There's something fishy here and I might end up finding it in the 
python2 codebase.

 

Is there any way you can see if you can reproduce this in the python2 codebase? 
Or, another thing that would be informative is if you can run the python3 unit 
tests in Windows and show the results? I don't have a Windows environment to 
test in.

> Getting error when reading avro message python
> --
>
> Key: AVRO-2591
> URL: https://issues.apache.org/jira/browse/AVRO-2591
> Project: Apache Avro
>  Issue Type: Bug
>  Components: python
>Affects Versions: 1.9.1
> Environment: Windows
>Reporter: Alperen Koksel
>Priority: Major
>
> I can't read avro messages in python in windows environment. 
> We 're using avro templates on kafka. I get the avro serialized message using 
> kafkaconsumer but I can't deserialize using avro.io.
>  
> {code:java}
> // code placeholder
> schema_path = "x.avsc"
> schema = avro.schema.Parse(open(schema_path).read())
> def from_avro(msg):
> bytes_reader = io.BytesIO(msg)
> decoder = avro.io.BinaryDecoder(bytes_reader)
> reader = avro.io.DatumReader(schema, schema)
> user1 = reader.read(decoder)
> return user1
> consumer = KafkaConsumer('avro_topic',
>  bootstrap_servers=['0.dual.kafka.qa-us.com:9092',
> '1.dual.kafka.qa-us.com:9092',
> '2.dual.kafka.qa-us.com:9092'],
>  auto_offset_reset='earliest',
>  group_id='my_group',
>  value_deserializer=from_avro
>  )
> for msg in consumer:
> print(msg){code}
>  
> {code:java}
> // code placeholder
> File 
> "C:\Users\alpl\PycharmProjects\WorkOn\venv\lib\site-packages\avro\io.py", 
> line 240, in read_bytes
>  assert (nbytes >= 0), nbytes
> AssertionError: -45
> {code}
>  
> I found following stackoverflow post which posted more than a year ago.
> [https://stackoverflow.com/questions/50106662/error-reading-avro-file-in-python/58336209#58336209]
> I tried several versions of avro. Results are same.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (AVRO-2591) Getting error when reading avro message python

[Michael A. Smith (Jira)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael A. Smith reassigned AVRO-2591:
--

Assignee: (was: Michael A. Smith)

> Getting error when reading avro message python
> --
>
> Key: AVRO-2591
> URL: https://issues.apache.org/jira/browse/AVRO-2591
> Project: Apache Avro
>  Issue Type: Bug
>  Components: python
>Affects Versions: 1.9.1
> Environment: Windows
>Reporter: Alperen Koksel
>Priority: Major
>
> I can't read avro messages in python in windows environment. 
> We 're using avro templates on kafka. I get the avro serialized message using 
> kafkaconsumer but I can't deserialize using avro.io.
>  
> {code:java}
> // code placeholder
> schema_path = "x.avsc"
> schema = avro.schema.Parse(open(schema_path).read())
> def from_avro(msg):
> bytes_reader = io.BytesIO(msg)
> decoder = avro.io.BinaryDecoder(bytes_reader)
> reader = avro.io.DatumReader(schema, schema)
> user1 = reader.read(decoder)
> return user1
> consumer = KafkaConsumer('avro_topic',
>  bootstrap_servers=['0.dual.kafka.qa-us.com:9092',
> '1.dual.kafka.qa-us.com:9092',
> '2.dual.kafka.qa-us.com:9092'],
>  auto_offset_reset='earliest',
>  group_id='my_group',
>  value_deserializer=from_avro
>  )
> for msg in consumer:
> print(msg){code}
>  
> {code:java}
> // code placeholder
> File 
> "C:\Users\alpl\PycharmProjects\WorkOn\venv\lib\site-packages\avro\io.py", 
> line 240, in read_bytes
>  assert (nbytes >= 0), nbytes
> AssertionError: -45
> {code}
>  
> I found following stackoverflow post which posted more than a year ago.
> [https://stackoverflow.com/questions/50106662/error-reading-avro-file-in-python/58336209#58336209]
> I tried several versions of avro. Results are same.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (AVRO-2591) Getting error when reading avro message python

[Michael A. Smith (Jira)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael A. Smith reassigned AVRO-2591:
--

Assignee: Michael A. Smith

> Getting error when reading avro message python
> --
>
> Key: AVRO-2591
> URL: https://issues.apache.org/jira/browse/AVRO-2591
> Project: Apache Avro
>  Issue Type: Bug
>  Components: python
>Affects Versions: 1.9.1
> Environment: Windows
>Reporter: Alperen Koksel
>Assignee: Michael A. Smith
>Priority: Major
>
> I can't read avro messages in python in windows environment. 
> We 're using avro templates on kafka. I get the avro serialized message using 
> kafkaconsumer but I can't deserialize using avro.io.
>  
> {code:java}
> // code placeholder
> schema_path = "x.avsc"
> schema = avro.schema.Parse(open(schema_path).read())
> def from_avro(msg):
> bytes_reader = io.BytesIO(msg)
> decoder = avro.io.BinaryDecoder(bytes_reader)
> reader = avro.io.DatumReader(schema, schema)
> user1 = reader.read(decoder)
> return user1
> consumer = KafkaConsumer('avro_topic',
>  bootstrap_servers=['0.dual.kafka.qa-us.com:9092',
> '1.dual.kafka.qa-us.com:9092',
> '2.dual.kafka.qa-us.com:9092'],
>  auto_offset_reset='earliest',
>  group_id='my_group',
>  value_deserializer=from_avro
>  )
> for msg in consumer:
> print(msg){code}
>  
> {code:java}
> // code placeholder
> File 
> "C:\Users\alpl\PycharmProjects\WorkOn\venv\lib\site-packages\avro\io.py", 
> line 240, in read_bytes
>  assert (nbytes >= 0), nbytes
> AssertionError: -45
> {code}
>  
> I found following stackoverflow post which posted more than a year ago.
> [https://stackoverflow.com/questions/50106662/error-reading-avro-file-in-python/58336209#58336209]
> I tried several versions of avro. Results are same.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] new committer Michael Smith

[Brian Lachniet]

Congrats and welcome!

On Sun, Oct 20, 2019 at 4:51 AM Driesprong, Fokko 
wrote:

> Welcome Michael, great to have you as a committer. You've done some awesome
> work on the Python part of Avro!
>
> Cheers, Fokko
>
> Op zo 20 okt. 2019 om 10:17 schreef Nándor Kollár :
>
> > Congrats, welcome Michael on board!
> >
> > On 2019/10/20 03:20:14, Sean Busbey  wrote:
> > > Hi folks!
> > >
> > > I'm very pleased to announce that Michael Smith has accepted the PMC's
> > > invitation to become a committer on the Apache Avro project.
> > >
> > > Please join me in congratulating Michael on this recognition of their
> > > great work thus far in our community.
> > >
> >
>


-- 

[image: 51b630b05e01a6d5134ccfd520f547c4.png]

Brian Lachniet

Software Engineer

E: blachn...@gmail.com | blachniet.com 

 

Re: [ANNOUNCE] new committer Michael Smith

[Driesprong, Fokko]

Welcome Michael, great to have you as a committer. You've done some awesome
work on the Python part of Avro!

Cheers, Fokko

Op zo 20 okt. 2019 om 10:17 schreef Nándor Kollár :

> Congrats, welcome Michael on board!
>
> On 2019/10/20 03:20:14, Sean Busbey  wrote:
> > Hi folks!
> >
> > I'm very pleased to announce that Michael Smith has accepted the PMC's
> > invitation to become a committer on the Apache Avro project.
> >
> > Please join me in congratulating Michael on this recognition of their
> > great work thus far in our community.
> >
>

Re: [ANNOUNCE] new committer Michael Smith

[Nándor Kollár]

Congrats, welcome Michael on board!

On 2019/10/20 03:20:14, Sean Busbey  wrote: 
> Hi folks!
> 
> I'm very pleased to announce that Michael Smith has accepted the PMC's
> invitation to become a committer on the Apache Avro project.
> 
> Please join me in congratulating Michael on this recognition of their
> great work thus far in our community.
>