Re: [Python-Dev] bitwise operations for bytes and bytearray

[Andrew Barnert via Python-Dev]

On Jan 9, 2016, at 16:17, Blake Griffith  wrote:
> 
> A little update, I got ^, &, and | working for bytearrays. You can view the 
> diff here:
> https://github.com/python/cpython/compare/master...cowlicks:bitwise-bytes?expand=1

If you upload the diff to the issue on the tracker, the reitveld code review 
app should be able to pick it up automatically, allowing people to comment on 
it inline, in a much nicer format than a mailing list thread. It's especially 
nice if you're adding things in stages--people who have been following along 
can just look at the changes between patch 3 and 4, while new people can look 
at all the changes in one go, etc.

> How does it look? 
> Joe, is this how I should allocate the arrays? Am I freeing them properly?
> Am I checking the input enough?
> 
> After some feedback, I'll probably add bitshifting and negation. Then work on 
> bytes objects.
> 
> Does this warrant a pep?

Personally, I'd just make the case for the feature on the tracker issue. If one 
of the core devs thinks it needs a PEP, or further discussion on this list or 
-ideas, they'll say so there.

At present, it seems like there's not much support for the idea, but I think 
that's at least partly because people want to see realistic use cases (that 
aren't server better by the existing bitarray/bitstring/etc. modules on PyPI, 
or using a NumPy array, or just using ints, etc.).

>> On Fri, Jan 8, 2016 at 2:08 AM, Cameron Simpson  wrote:
>>> On 07Jan2016 16:12, Python-Dev  wrote:
>>> On Jan 7, 2016, at 15:57, Martin Panter  wrote:
> On 7 January 2016 at 22:26, Blake Griffith  
> wrote:
> I'm interested in adding the functionality to do something like:
 b'a' ^ b'b'
> b'\x03'
> Instead of the good ol' TypeError.
> 
> I think both bytes and bytearray should support all the bitwise 
> operations.
 
 There is a bug open about adding this kind of functionality:
 .
>>> 
>>> And it's in the needs patch stage, which makes it perfect for the OP: in 
>>> addition to learning how to hack on builtin types, he can also learn the 
>>> other parts of the dev process. (Even if the bug is eventually rejected, as 
>>> seems likely given that it sat around for three years with no compelling 
>>> use case  and then Guido added a "very skeptical" comment.)
>> 
>> The use case which springs immediately to my mind is cryptography. To 
>> encrypt a stream symmetrically you can go:
>> 
>>  cleartext-bytes ^ cryptographicly-random-bytes-from-cipher
>> 
>> so with this one could write:
>> 
>>  def crypted(byteses, crypto_source):
>>''' Accept an iterable source of bytes objects and a preprimed source of  
>>   crypto bytes, yield encrypted versions of the bytes objects.
>>'''
>>for bs in byteses:
>>  cbs = crypto_source.next_bytes(len(bs))
>>  yield bs ^ cbs
>> 
>> Cheers,
>> Cameron Simpson 
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>> https://mail.python.org/mailman/options/python-dev/blake.a.griffith%40gmail.com
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bitwise operations for bytes and bytearray

[Blake Griffith]

A little update, I got ^, &, and | working for bytearrays. You can view the
diff here:
https://github.com/python/cpython/compare/master...cowlicks:bitwise-bytes?expand=1

How does it look?
Joe, is this how I should allocate the arrays? Am I freeing them properly?
Am I checking the input enough?

After some feedback, I'll probably add bitshifting and negation. Then work
on bytes objects.

Does this warrant a pep?

On Fri, Jan 8, 2016 at 2:08 AM, Cameron Simpson  wrote:

> On 07Jan2016 16:12, Python-Dev  wrote:
>
>> On Jan 7, 2016, at 15:57, Martin Panter  wrote:
>>
>>> On 7 January 2016 at 22:26, Blake Griffith 
 wrote:
 I'm interested in adding the functionality to do something like:

> b'a' ^ b'b'
>>>
>> b'\x03'
 Instead of the good ol' TypeError.

 I think both bytes and bytearray should support all the bitwise
 operations.

>>>
>>> There is a bug open about adding this kind of functionality:
>>> .
>>>
>>
>> And it's in the needs patch stage, which makes it perfect for the OP: in
>> addition to learning how to hack on builtin types, he can also learn the
>> other parts of the dev process. (Even if the bug is eventually rejected, as
>> seems likely given that it sat around for three years with no compelling
>> use case  and then Guido added a "very skeptical" comment.)
>>
>
> The use case which springs immediately to my mind is cryptography. To
> encrypt a stream symmetrically you can go:
>
>  cleartext-bytes ^ cryptographicly-random-bytes-from-cipher
>
> so with this one could write:
>
>  def crypted(byteses, crypto_source):
>''' Accept an iterable source of bytes objects and a preprimed source
> ofcrypto bytes, yield encrypted versions of the bytes objects.
>'''
>for bs in byteses:
>  cbs = crypto_source.next_bytes(len(bs))
>  yield bs ^ cbs
>
> Cheers,
> Cameron Simpson 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/blake.a.griffith%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bitwise operations for bytes and bytearray

[Martin Panter]

On 10 January 2016 at 00:17, Blake Griffith  wrote:
> A little update, I got ^, &, and | working for bytearrays. You can view the
> diff here:
> https://github.com/python/cpython/compare/master...cowlicks:bitwise-bytes?expand=1
>
> How does it look?

I left some comments against your commits on Git Hub. Unfortunately
they seem to be made in the base Python repository, rather than your
fork, which I did not expect.

> Joe, is this how I should allocate the arrays? Am I freeing them properly?
> Am I checking the input enough?
>
> After some feedback, I'll probably add bitshifting and negation. Then work
> on bytes objects.
>
> Does this warrant a pep?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bitwise operations for bytes and bytearray

[Joe Jevnik]

You want to put the `xor` method in the `nb_xor` field of the
`PyNumberMethods` structure that lives in the `tp_as_number` field of the
bytes type object. Two things I noticed in a quick pass: you might want to
add some type checking around the case where `a` or `b` is not a
`PyByteArray` object. Also, variable length arrays are added in C99, so you
will need to manually allocate the `raw_*` arrays in the heap. This seems
like a cool feature!

On Thu, Jan 7, 2016 at 5:26 PM, Blake Griffith 
wrote:

> Hi!
>
> I'm interested in adding the functionality to do something like:
>
> >>>  b'a' ^ b'b'
> b'\x03'
>
>
> Instead of the good ol' TypeError.
>
> I think both bytes and bytearray should support all the bitwise operations.
>
> I've never hacked on cpython before. I'm starting by just trying to add
> xor to bytearray. I have a ByteArray_Xor function that I think should do
> this here
> https://github.com/cowlicks/cpython/commit/d6dddb11cdb33032b39dcb9dfdaa7b10d4377b5f
>
> But I'm not sure how to hook this in to the rest of cypython. I tried
> adding it where bytearray_as_sequence is declared in this
> bytearrayobject.c file. But that gave me compiler warnings and broke things.
>
> So now that I have this ByteArray_Xor function, how do I make it be
> bytearray.__xor___?
>
> Thanks!
>
> Blake
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/joe%40quantopian.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bitwise operations for bytes and bytearray

[Andrew Barnert via Python-Dev]

On Jan 7, 2016, at 15:57, Martin Panter  wrote:
> 
>> On 7 January 2016 at 22:26, Blake Griffith  
>> wrote:
>> I'm interested in adding the functionality to do something like:
>> 
> b'a' ^ b'b'
>> b'\x03'
>> 
>> 
>> Instead of the good ol' TypeError.
>> 
>> I think both bytes and bytearray should support all the bitwise operations.
> 
> There is a bug open about adding this kind of functionality:
> .

And it's in the needs patch stage, which makes it perfect for the OP: in 
addition to learning how to hack on builtin types, he can also learn the other 
parts of the dev process. (Even if the bug is eventually rejected, as seems 
likely given that it sat around for three years with no compelling use case  
and then Guido added a "very skeptical" comment.)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bitwise operations for bytes and bytearray

[Blake Griffith]

Thanks for the quick responses y'all. I have something compiling on my
branch, which is enough for me tonight.

I asked a question about this on stackoverflow a while ago, it wasn't very
popular
https://stackoverflow.com/questions/32658420/why-cant-you-xor-bytes-objects-in-python

Someone there pointed out this feature was suggested on the mailing list a
while back (2006)
https://mail.python.org/pipermail/python-dev/2006-March/061980.html

On Fri, Jan 8, 2016 at 1:12 AM, Andrew Barnert  wrote:

> On Jan 7, 2016, at 15:57, Martin Panter  wrote:
> >
> >> On 7 January 2016 at 22:26, Blake Griffith 
> wrote:
> >> I'm interested in adding the functionality to do something like:
> >>
> > b'a' ^ b'b'
> >> b'\x03'
> >>
> >>
> >> Instead of the good ol' TypeError.
> >>
> >> I think both bytes and bytearray should support all the bitwise
> operations.
> >
> > There is a bug open about adding this kind of functionality:
> > .
>
> And it's in the needs patch stage, which makes it perfect for the OP: in
> addition to learning how to hack on builtin types, he can also learn the
> other parts of the dev process. (Even if the bug is eventually rejected, as
> seems likely given that it sat around for three years with no compelling
> use case  and then Guido added a "very skeptical" comment.)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bitwise operations for bytes and bytearray

[Martin Panter]

On 7 January 2016 at 22:26, Blake Griffith  wrote:
> I'm interested in adding the functionality to do something like:
>
  b'a' ^ b'b'
> b'\x03'
>
>
> Instead of the good ol' TypeError.
>
> I think both bytes and bytearray should support all the bitwise operations.

There is a bug open about adding this kind of functionality:
.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bitwise operations for bytes and bytearray

[Cameron Simpson]

On 07Jan2016 16:12, Python-Dev  wrote:

On Jan 7, 2016, at 15:57, Martin Panter  wrote:

On 7 January 2016 at 22:26, Blake Griffith  wrote:
I'm interested in adding the functionality to do something like:

b'a' ^ b'b'

b'\x03'
Instead of the good ol' TypeError.

I think both bytes and bytearray should support all the bitwise operations.


There is a bug open about adding this kind of functionality:
.


And it's in the needs patch stage, which makes it perfect for the OP: in 
addition to learning how to hack on builtin types, he can also learn the other 
parts of the dev process. (Even if the bug is eventually rejected, as seems 
likely given that it sat around for three years with no compelling use case  
and then Guido added a "very skeptical" comment.)


The use case which springs immediately to my mind is cryptography. To encrypt a 
stream symmetrically you can go:


 cleartext-bytes ^ cryptographicly-random-bytes-from-cipher

so with this one could write:

 def crypted(byteses, crypto_source):
   ''' Accept an iterable source of bytes objects and a preprimed source of 
   crypto bytes, yield encrypted versions of the bytes objects.

   '''
   for bs in byteses:
 cbs = crypto_source.next_bytes(len(bs))
 yield bs ^ cbs

Cheers,
Cameron Simpson 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] bitwise operations for bytes and bytearray

[Blake Griffith]

Hi!

I'm interested in adding the functionality to do something like:

>>>  b'a' ^ b'b'
b'\x03'


Instead of the good ol' TypeError.

I think both bytes and bytearray should support all the bitwise operations.

I've never hacked on cpython before. I'm starting by just trying to add xor
to bytearray. I have a ByteArray_Xor function that I think should do this
here
https://github.com/cowlicks/cpython/commit/d6dddb11cdb33032b39dcb9dfdaa7b10d4377b5f

But I'm not sure how to hook this in to the rest of cypython. I tried
adding it where bytearray_as_sequence is declared in this bytearrayobject.c
file. But that gave me compiler warnings and broke things.

So now that I have this ByteArray_Xor function, how do I make it be
bytearray.__xor___?

Thanks!

Blake
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bitwise operations for bytes and bytearray

[Brett Cannon]

On Thu, 7 Jan 2016 at 14:29 Blake Griffith 
wrote:

> Hi!
>
> I'm interested in adding the functionality to do something like:
>
> >>>  b'a' ^ b'b'
> b'\x03'
>
>
> Instead of the good ol' TypeError.
>
> I think both bytes and bytearray should support all the bitwise operations.
>
> I've never hacked on cpython before. I'm starting by just trying to add
> xor to bytearray. I have a ByteArray_Xor function that I think should do
> this here
> https://github.com/cowlicks/cpython/commit/d6dddb11cdb33032b39dcb9dfdaa7b10d4377b5f
>
> But I'm not sure how to hook this in to the rest of cypython. I tried
> adding it where bytearray_as_sequence is declared in this
> bytearrayobject.c file. But that gave me compiler warnings and broke things.
>
> So now that I have this ByteArray_Xor function, how do I make it be
> bytearray.__xor___?
>

You need to set the PyNumberMethods struct with the appropriate function
and then set that on the PyTypeObject. Look at
https://hg.python.org/cpython/file/tip/Include/object.h#l237 and
https://hg.python.org/cpython/file/tip/Objects/longobject.c#l5238 for an
idea of what it takes.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com