[issue46108] Enum repr() incorrect when mixed with non-__new__ data types

[Ethan Furman]

Ethan Furman  added the comment:

Fixed in 3.11.

--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed
superseder:  -> Enum: modify __repr__, __str__; update docs
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46108] Enum repr() incorrect when mixed with non-__new__ data types

[Ethan Furman]

Ethan Furman  added the comment:

Thanks, test added.  I also updated the title.

Code updated to treat any __repr__s as affecting the display of the member 
value instead of the member itself (i.e. the "<...: %r>" portion).  if a user 
actually wants to change the member repr they can assign it when creating the 
Enum:

class Entries(Foo, Enum):
ENTRY = 1
__repr__ = Foo.__repr__

assert repr(Entries.ENTRY1) == 'Entries(a=1)'

--
title: Enum repr() incorrect when mixed with dataclasses -> Enum repr() 
incorrect when mixed with non-__new__ data types

___
Python tracker 
<https://bugs.python.org/issue46108>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

ANNOUNCE: Thesaurus and ThesaurusCfg - recursive mapping and cfg file data types

[Dave Cinege]

This announcement is for a pre-release that I would like people to 
comment on structure, naming, etc. (Code review maybe not yet. :-)


Before you say "It's all been done before." I suggest you take a closer 
look and I think you may conclude that what I've revised over 7 years is 
now interesting.


---
Thesaurus is a mapping data type with key recursion and attribute 
aliasing. It is a subclass of dict() and compatible as a dictionary 
replacement baring where key path recursion may take place.


ThesaurusExtended is a subclass of Thesaurus providing additional 
usability methods.


ThesaurusCfg is a subclass of ThesaurusExtended providing a 
configuration file parser and per key data coercion methods.


The Thesaurus family works with Python 2.6+ to 3.8+.


A simple example of ThesaurusCfg
--
cfgs = '''
prog.version(static_int)= 123
opt.verbose (str_to_bool)   = yes
hi  = Hello
'''
from thesauruscfg import thescfg
cfg = thescfg()
>>> cfg.parse(cfgs)
{'prog': {'version': 123}, 'opt': {'verbose': True}, 'hi': 'Hello'}
>>> cfg.opt.verbose
True

import json
>>> print(json.dumps(cfg, indent=4, separators=(',', ': ')))
{
"prog": {
"version": 123
},
"opt": {
"verbose": true
},
"hi": "Hello"
}

browse:
  https://git.cinege.com/thesaurus/
or
  git clone https://git.cinege.com/thesaurus/
---

Dave
--
https://mail.python.org/mailman/listinfo/python-list

[issue33602] Remove set and queue references from Data Types

[Julien Palard]

Change by Julien Palard :


--
nosy:  -mdk
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33602] Remove set and queue references from Data Types

[Julien Palard]

Julien Palard  added the comment:


New changeset 816825e192ed430438c613d52a58fb9e1a8d90f4 by Julien Palard (Miss 
Islington (bot)) in branch '3.7':
bpo-33602: Doc: Remove set and queue references from Data Types (GH-7055) 
(GH-15876)
https://github.com/python/cpython/commit/816825e192ed430438c613d52a58fb9e1a8d90f4


--

___
Python tracker 
<https://bugs.python.org/issue33602>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33602] Remove set and queue references from Data Types

[Julien Palard]

Julien Palard  added the comment:


New changeset 58ef7d341c79f649da275bb1d5c11f668d7bac9e by Julien Palard (Miss 
Islington (bot)) in branch '3.8':
bpo-33602: Doc: Remove set and queue references from Data Types (GH-7055) 
(GH-15875)
https://github.com/python/cpython/commit/58ef7d341c79f649da275bb1d5c11f668d7bac9e


--

___
Python tracker 
<https://bugs.python.org/issue33602>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33602] Remove set and queue references from Data Types

[miss-islington]

Change by miss-islington :


--
pull_requests: +15516
pull_request: https://github.com/python/cpython/pull/15876

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33602] Remove set and queue references from Data Types

[miss-islington]

Change by miss-islington :


--
pull_requests: +15515
pull_request: https://github.com/python/cpython/pull/15875

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33602] Remove set and queue references from Data Types

[Julien Palard]

Julien Palard  added the comment:


New changeset 912108891db52c2067889be1f4ce5713839807cd by Julien Palard (Andre 
Delfino) in branch 'master':
bpo-33602: Doc: Remove set and queue references from Data Types (GH-7055)
https://github.com/python/cpython/commit/912108891db52c2067889be1f4ce5713839807cd


--
nosy: +mdk

___
Python tracker 
<https://bugs.python.org/issue33602>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: What data types does matplotlib pyplot take?

[Peter Pearson]

On Wed, 13 Jun 2018 12:53:57 -0400, C W  wrote:
> Hi everyone,
>
> I'm curious what data types pyplot takes. It seems that it can take numpy
> series, pandas series, and possibly pandas dataframe? How many people data
> types are out there? Is that true for all functions in like hist(), bar(),
> line(), etc?

I regret that this might seem less than helpful, but someone
should point out that an enumeration beginning "This package
works with the following data types" would be antithetical to
Python's "duck-typing" philosophy.  The important thing is not
an object's type, but whether the object has the attributes and
methods required.

-- 
To email me, substitute nowhere->runbox, invalid->com.
-- 
https://mail.python.org/mailman/listinfo/python-list

What data types does matplotlib pyplot take?

[C W]

Hi everyone,

I'm curious what data types pyplot takes. It seems that it can take numpy
series, pandas series, and possibly pandas dataframe? How many people data
types are out there? Is that true for all functions in like hist(), bar(),
line(), etc?

Is there an official documentation that lists this?

I have seen this official documentation:
https://pandas.pydata.org/pandas-docs/stable/visualization.html

But it does not specify data types. I would appreciate it if someone can
point it out.

Thank you!
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue33602] Remove set and queue references from Data Types

[Andrés Delfino]

Change by Andrés Delfino :


--
keywords: +patch
pull_requests: +6689
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33602] Remove set and queue references from Data Types

[Andrés Delfino]

New submission from Andrés Delfino <adelf...@gmail.com>:

Data Types mentions sets (which are now built-in) and synchronized queues (now 
mentioned in Concurrent Execution). I'm proposing fixing this.

PR also adds mention to bytearray.

--
assignee: docs@python
components: Documentation
messages: 317293
nosy: adelfino, docs@python
priority: normal
severity: normal
status: open
title: Remove set and queue references from Data Types
type: enhancement
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue33602>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Learning python data types

[Rustom Mody]

On Monday, October 3, 2016 at 3:38:24 AM UTC+5:30, Crane Ugly wrote:
> I am not new in scripting in general.
> And about a year I create scripts in python, but my approach was always based 
> on my habits from UNIX shell, where data types are not obvious.
> Going deeper with python I see necessity to know better data types, 
> differences between them and most importantly when one data type if better 
> then the other, with life examples.
> Unfortunately, books I've see so far describe data types in very formal way 
> and without taking into account day-to-day practical aspects.
> For example in my practice data rarely hardcoded in the program code and come 
> from files, streams or generated in the program. How to put them together for 
> better processing?
> I need some good examples of books or web articles, that can describe 
> language from data processing perspective, not just academical explanations 
> of when language is.
> 
> Leonid

Yes this is a real need and very badly satisfied at the moment

- Language texts are language centered — mostly what you'll get out here
- Data Structure texts tend to emphasize storage structures — eg 
  ‘linked-lists’ — rather than data structures eg sets, bags, lists and their 
  complementary semantics
- And the academic texts are too academic/theoretical

Something I had written [Sorry! very much in the 3rd category!!] nearly two 
decades ago:
http://cs.unipune.ac.in/admin/obx/hod/course.pdf
particularly the Intro to programming, discrete structures and data structures
syllabi
-- 
https://mail.python.org/mailman/listinfo/python-list

Learning python data types

[Crane Ugly]

I am not new in scripting in general.
And about a year I create scripts in python, but my approach was always based 
on my habits from UNIX shell, where data types are not obvious.
Going deeper with python I see necessity to know better data types, differences 
between them and most importantly when one data type if better then the other, 
with life examples.
Unfortunately, books I've see so far describe data types in very formal way and 
without taking into account day-to-day practical aspects.
For example in my practice data rarely hardcoded in the program code and come 
from files, streams or generated in the program. How to put them together for 
better processing?
I need some good examples of books or web articles, that can describe language 
from data processing perspective, not just academical explanations of when 
language is.

Leonid
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Data Types

[Chris Angelico]

On Thu, Sep 22, 2016 at 10:33 PM, BartC  wrote:
>>>   print (10<20)=>  True
>>>   print (type(10<20))  =>  
>>
>>
>> 10<20 shouldn't be thought of as some alternative value which is a bool,
>> any
>> more than we should think of 1+1 as being a different value to 2.
>
>
> They're a little different:
>
>  1+1 yielding 2 is  int+int => int
>  10<20 yielding True is int bool

That's a couple of example expressions and the way they're handled.

> My post was really about bool values lurking everywhere not just where you
> explicitly write or assign True or False.

Sure. Thing is, those expressions are actually just syntactic sugar
for functions. Let's replace the integers with these puppies:

class Spam:
def __init__(self, value):
self.value = value
def __repr__(self):
return "Spam(%r)" % self.value
def __add__(self, other):
return Spam(self.value + other.value)
def __lt__(self, other):
if self.value < other.value:
return "Yes, %r < %r" % (self.value, other.value)
return ""

>>> Spam(1) + Spam(1)
Spam(2)
>>> Spam(10) < Spam(20)
'Yes, 10 < 20'
>>> Spam(10) > Spam(20)
''

This is perfectly legal code, and it doesn't use True or False for its
comparisons. If you want it to, you have to actually return one of
those constants from __lt__, either by explicitly typing its name, or
by passing it up the chain (eg "return self.value < other.value"),
which just moves the problem around.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Data Types

[BartC]

On 22/09/2016 02:40, Steve D'Aprano wrote:

On Wed, 21 Sep 2016 10:25 pm, BartC wrote:


On 21/09/2016 05:03, Cai Gengyang wrote:


Are there any other data types that will give you type(A) or type(B) =
 besides True and False?


No types but any variable or expression containing True or False will be
a bool type (or class bool):


"Containing" True or False? Certainly not:


Well, I wrote 'containing' before I added 'or expression', when 
'evaluating to' might have been better.



  print (10<20)=>  True
  print (type(10<20))  =>  


10<20 shouldn't be thought of as some alternative value which is a bool, any
more than we should think of 1+1 as being a different value to 2.


They're a little different:

 1+1 yielding 2 is  int+int => int
 10<20 yielding True is int bool

My post was really about bool values lurking everywhere not just where 
you explicitly write or assign True or False.



--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Data Types

[Steve D'Aprano]

On Thu, 22 Sep 2016 01:46 pm, Sayth Renshaw wrote:

> What about 0 or 1 they are true and false like no other numbers? what
> category do they fall in with regards to booleans?

0 and 1 are ints:

py> type(0)

py> type(1)



just like 2, 3, 4, -1, -999, 1098765432 etc.

But just like 1 == 1.0 (a float), and 0 == 0.0 (a float), and in fact 1 also
equals Fraction(1, 1) and Decimal(1) and 1+0j (complex number), so 1 also
equals the bool True and 0 equals the bool False.

The reason for this is that in many programming languages, there are no
boolean values. People use 0 for false and 1 (or sometimes -1) for true. In
the early days, Python was the same.

See https://www.python.org/dev/peps/pep-0285/ for more details.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Data Types

[Lawrence D’Oliveiro]

On Thursday, September 22, 2016 at 3:47:27 PM UTC+12, Sayth Renshaw wrote:
> What about 0 or 1 they are true and false like no other numbers?

>>> isinstance(1, bool)
False
>>> isinstance(True, int)
True
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Data Types

[Larry Hudson via Python-list]

On 09/20/2016 09:03 PM, Cai Gengyang wrote:
[snip...]

So for example for "bool" , it only applies to True/False and nothing else? (2 
data types), i.e. :


Not exactly...  bool is the data type (or class), True and False are the only two _values_ (not 
types).





type(True)



type(False)




[snip...]

Besides the bool class, with its two values, there is also the Nonetype class, which has only 
one value:  None.


But to expand on/rephrase what has already been said:  while the bool class/type has only the 
two values True/False, in expressions using logical operators (and, or, not...), ANY data type 
can be interpreted as a boolean.  These are often referred to as truthy and falsey values to 
distinguish them from actual boolean True/False values.


I'm sure you've already seen the list, if not you should quickly learn it...
Any numeric (int, float, complex) that is zero is falsey.
Any empty collection (list, string, dictionary...) is falsey.
EVERYTHING else is truthy.

--
 -=- Larry -=-
--
https://mail.python.org/mailman/listinfo/python-list

Re: Data Types

[Sayth Renshaw]

> > 
> >> Are there any other data types that will give you type(A) or type(B) =
> >>  besides True and False?
> > 
> > No types but any variable or expression containing True or False will be
> > a bool type (or class bool):
> 
> "Containing" True or False? Certainly not:
> 
> py> type( [1, 2, True] )
> 
> py> type( False or 99 )
> 
> 
> Based on your examples, I think you mean any expression *evaluating to* True
> or False will be a bool. Um, yeah, of course it will. Because it evaluates
> to one of the only two bool values, True or False.
> 
> >   A = 10<20
> >   print (type(A))  =>  
> 
> That's because the value of A is True.
> 
> >   print (10<20)=>  True
> >   print (type(10<20))  =>  
> 
> 10<20 shouldn't be thought of as some alternative value which is a bool, any
> more than we should think of 1+1 as being a different value to 2.
> 
> 

What about 0 or 1 they are true and false like no other numbers? what category 
do they fall in with regards to booleans?

In [6]: 0 == False
Out[6]: True

In [7]: 1 == True
Out[7]: True

In [8]: 2 == True
Out[8]: False

In [9]: 3 == True
Out[9]: False

In [10]: 3 == False
Out[10]: False

Sayth
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Data Types

[Steve D'Aprano]

On Wed, 21 Sep 2016 10:25 pm, BartC wrote:

> On 21/09/2016 05:03, Cai Gengyang wrote:
> 
>> Are there any other data types that will give you type(A) or type(B) =
>>  besides True and False?
> 
> No types but any variable or expression containing True or False will be
> a bool type (or class bool):

"Containing" True or False? Certainly not:

py> type( [1, 2, True] )

py> type( False or 99 )


Based on your examples, I think you mean any expression *evaluating to* True
or False will be a bool. Um, yeah, of course it will. Because it evaluates
to one of the only two bool values, True or False.

>   A = 10<20
>   print (type(A))  =>  

That's because the value of A is True.

>   print (10<20)=>  True
>   print (type(10<20))  =>  

10<20 shouldn't be thought of as some alternative value which is a bool, any
more than we should think of 1+1 as being a different value to 2.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Data Types

[BartC]

On 21/09/2016 05:03, Cai Gengyang wrote:


Are there any other data types that will give you type(A) or type(B) =  besides True and False?


No types but any variable or expression containing True or False will be 
a bool type (or class bool):


 A = 10<20
 print (type(A))  =>  

 print (10<20)=>  True

 print (type(10<20))  =>  

 --
Bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: Data Types

[Steven D'Aprano]

On Wednesday 21 September 2016 14:03, Cai Gengyang wrote:

> So for example for "bool" , it only applies to True/False and nothing else?

Correct. True and False are the only instances of the type 'bool'.

> (2 data types), i.e. :
> 
>>>> type(True)
> 
>>>> type(False)
> 
> 
> Are there any other data types that will give you type(A) or type(B) =  'bool'> besides True and False?

No.





-- 
Steven
git gets easier once you get the basic idea that branches are homeomorphic 
endofunctors mapping submanifolds of a Hilbert space.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Data Types

[Cai Gengyang]

On Wednesday, September 21, 2016 at 11:22:42 AM UTC+8, Steve D'Aprano wrote:
> On Wed, 21 Sep 2016 01:00 pm, Cai Gengyang wrote:
> 
> > So I am going through chapter 7.1 (Data Types) of
> > http://programarcadegames.com/index.php… What do these terms (input and
> > output) mean --- Can someone explain in plain English and layman's terms ?
> 
> http://www.dictionary.com/browse/input
> 
> http://www.dictionary.com/browse/output
> 
> 
> Does that answer your question about "input and output"?
> 
> 
> > Thanks a lot ...
> > 
> >>>> type(3)
> > 
> >>>> type(3.145)
> > 
> >>>> type("Hi there")
> > 
> >>>> type(True)
> > 
> 
> I don't understand why you are showing this. What part don't you understand?
> 
> The number 3 is an int (short for integer); that is the type of value it is.
> 5 is also an int, and 0, and 9253, and -73. They are all integers.
> 
> 3.145 is a float (short for "floating point number"). "Hi there" is a str
> (short for string). True is a bool (short for Boolean value, which is a
> technical term for special True/False values).
> 
> Perhaps if you can ask a more clear question, I can give a more clear
> answer.
> 
> 
> 
> 
> -- 
> Steve
> “Cheer up,” they said, “things could be worse.” So I cheered up, and sure
> enough, things got worse.

Right, 

So for example for "bool" , it only applies to True/False and nothing else? (2 
data types), i.e. :

>>> type(True)

>>> type(False)


Are there any other data types that will give you type(A) or type(B) =  besides True and False?

Regards,

GY

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Data Types

[Steve D'Aprano]

On Wed, 21 Sep 2016 01:00 pm, Cai Gengyang wrote:

> So I am going through chapter 7.1 (Data Types) of
> http://programarcadegames.com/index.php… What do these terms (input and
> output) mean --- Can someone explain in plain English and layman's terms ?

http://www.dictionary.com/browse/input

http://www.dictionary.com/browse/output


Does that answer your question about "input and output"?


> Thanks a lot ...
> 
>>>> type(3)
> 
>>>> type(3.145)
> 
>>>> type("Hi there")
> 
>>>> type(True)
> 

I don't understand why you are showing this. What part don't you understand?

The number 3 is an int (short for integer); that is the type of value it is.
5 is also an int, and 0, and 9253, and -73. They are all integers.

3.145 is a float (short for "floating point number"). "Hi there" is a str
(short for string). True is a bool (short for Boolean value, which is a
technical term for special True/False values).

Perhaps if you can ask a more clear question, I can give a more clear
answer.




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Data Types

[Cai Gengyang]

So I am going through chapter 7.1 (Data Types) of 
http://programarcadegames.com/index.php…
What do these terms (input and output) mean --- Can someone explain in plain 
English and layman's terms ? Thanks a lot ...

>>> type(3)

>>> type(3.145)

>>> type("Hi there")

>>> type(True)

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Best way to deal with different data types in a list comprehension

[Steven D'Aprano]

Larry Martell wrote:

 I have some code that I inherited:
 
 ' '.join([self.get_abbrev()] +
[str(f['value')
 for f in self.filters
 if f.has_key('value')]).strip()
 
 
 This broke today when it encountered some non-ascii data.

It's already broken. It gives a Syntax Error part way through:

py ' '.join([self.get_abbrev()] +
...[str(f['value')
  File stdin, line 2
[str(f['value')
  ^
SyntaxError: invalid syntax

Please copy and paste the actual code, don't retype it.

This is my guess of what you actually have, reformatted to make it more
clear (at least to me):

' '.join(
[self.get_abbrev()] + 
[str(f['value']) for f in self.filters if f.has_key('value')]
).strip()

I *think* that the call to strip() is redundant. Hmmm... perhaps not, if the
self.get_abbrev() begins with whitespace, or the last f['value'] ends with
whitespace. You should consider removing that call to .strip(), but for now
I'll assume it actually is useful and leave it in.

First change: assuming the filters are dicts, do the test this way:

' '.join(
[self.get_abbrev()] + 
[str(f['value']) for f in self.filters if 'value' in f]
).strip()


Now, the *right* way to fix your problem is to convert the whole application
to use unicode strings everywhere instead of byte strings. I'm guessing you
are using Python 2.6 or 2.7.

You say it broke when given some non-ascii data, but that's extremely
ambiguous. {23: 42} is non-ascii data. What exactly do you have, and where
did it come from?

My *guess* is that you had a Unicode string, containing characters which
cannot be converted to ASCII.

py str(u'Ωπ')
Traceback (most recent call last):
  File stdin, line 1, in module
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1:
ordinal not in range(128)


 I changed the str(f['value']) line to f['value'].encode('utf-8'),

Hmmm, I'm not sure that's a good plan:

py u'Ωπ'.encode('utf-8')
'\xce\xa9\xcf\x80'

Do you really want to find arbitrary bytes floating through your strings? A
better strategy is to convert the program to use unicode strings
internally, and only convert to byte strings when you read and write to
files.

But assuming you don't have the time or budget for that sort of re-write,
here's a minimal chance which might do the job:

u' '.join(
[self.get_abbrev()] + 
[unicode(f['value']) for f in self.filters if 'value' in f]
).strip()


That works correctly for random objects and ASCII byte strings:

py unicode([1, 2, 3])
u'[1, 2, 3]'
py unicode('bytes')
u'bytes'


Alas, it will fail for non-ASCII byte strings:

py unicode('bytes \xFF')
Traceback (most recent call last):
  File stdin, line 1, in module
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 6:
ordinal not in range(128)


Here's a version which prefers byte-strings, but should be able to handle
everything you throw at it:

' '.join(
  [self.get_abbrev()] + 
  [
   (x.encode('utf-8') if isinstance(x, unicode) else x) 
   for x in (f['value'] for f in self.filters if 'value' in f)
  ]
).strip()



Note the use of a generator expression inside the list comp.


-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Best way to deal with different data types in a list comprehension

[Larry Martell]

I have some code that I inherited:

' '.join([self.get_abbrev()] +
   [str(f['value')
for f in self.filters
if f.has_key('value')]).strip()


This broke today when it encountered some non-ascii data.

I changed the str(f['value']) line to f['value'].encode('utf-8'),
which works fine, except when f['value'] is not a string (it could be
anything).

Without rewriting this without the list comprehension, how can I write
this to deal with both strings and non-strings?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Best way to deal with different data types in a list comprehension

[Rock Neurotiko]

Maybe there are a different way, but you can do this:

' '.join([self.get_abbrev()] +
   [str(f['value').encode('utf-8') if type(f['value']) is str else
str(f['value']
for f in self.filters
if f.has_key('value')]).strip()

2014-09-24 0:01 GMT+02:00 Larry Martell larry.mart...@gmail.com:

 I have some code that I inherited:

 ' '.join([self.get_abbrev()] +
[str(f['value')
 for f in self.filters
 if f.has_key('value')]).strip()


 This broke today when it encountered some non-ascii data.

 I changed the str(f['value']) line to f['value'].encode('utf-8'),
 which works fine, except when f['value'] is not a string (it could be
 anything).

 Without rewriting this without the list comprehension, how can I write
 this to deal with both strings and non-strings?
 --
 https://mail.python.org/mailman/listinfo/python-list




-- 
Miguel García Lafuente - Rock Neurotiko

Do it, the devil is in the details.
The quieter you are, the more you are able to hear.
Happy Coding. Code with Passion, Decode with Patience.
If we make consistent effort, based on proper education, we can change the
world.

El contenido de este e-mail es privado, no se permite la revelacion del
contenido de este e-mail a gente ajena a él.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Best way to deal with different data types in a list comprehension

[Chris Kaynor]

On Tue, Sep 23, 2014 at 3:01 PM, Larry Martell larry.mart...@gmail.com
wrote:

 I have some code that I inherited:

 ' '.join([self.get_abbrev()] +
[str(f['value')
 for f in self.filters
 if f.has_key('value')]).strip()


 This broke today when it encountered some non-ascii data.


One option would be to do the processing in unicode, and convert to utf-8
only when needed:

u' '.join([self.get_abbrev()] +
   [unicode(f['value')
for f in self.filters
if f.has_key('value')]).strip()

If needed, add a .encode('utf-8') to the end.



 I changed the str(f['value']) line to f['value'].encode('utf-8'),
 which works fine, except when f['value'] is not a string (it could be
 anything).

 Without rewriting this without the list comprehension, how can I write
 this to deal with both strings and non-strings?
 --
 https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Best way to deal with different data types in a list comprehension

[Larry Martell]

On Tue, Sep 23, 2014 at 6:05 PM, Rock Neurotiko
miguelglafue...@gmail.com wrote:
 2014-09-24 0:01 GMT+02:00 Larry Martell larry.mart...@gmail.com:

 I have some code that I inherited:

 ' '.join([self.get_abbrev()] +
[str(f['value')
 for f in self.filters
 if f.has_key('value')]).strip()


 This broke today when it encountered some non-ascii data.

 I changed the str(f['value']) line to f['value'].encode('utf-8'),
 which works fine, except when f['value'] is not a string (it could be
 anything).

 Without rewriting this without the list comprehension, how can I write
 this to deal with both strings and non-strings?

 Maybe there are a different way, but you can do this:

 ' '.join([self.get_abbrev()] +
[str(f['value').encode('utf-8') if type(f['value']) is str else
 str(f['value']
 for f in self.filters
 if f.has_key('value')]).strip()

Thanks for the reply, but please don't top post.

This worked for me:

'.join([self.get_abbrev()] +
 [f['value'].encode('utf-8') if type(f['value']) is unicode
else str(f['value'])
  for f in self.filters
  if f.has_key('value')]).strip()
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: MySQL data types vs Django/Python data types

[Fábio Santos]

On 1 Sep 2013 21:54, Gary Roach gary719_li...@verizon.net wrote:

 Hi all,

 System:
 Debian Wheezy Linux
 Python 2.7
 Django 1.5
 MySql 5.5

 I am new to Python and Django and am having trouble matching Python data
types with those of MySQL. MySQL has about 7 basic data types including
Blobs, Binaries, etc. It also has a rich selection of geometry types. In
addition, types like INT have 7 or 8 different options like Primary Key,
zero fill, auto inc, etc. I can't seem to find anything in python to match
these. I am trying to build a model.py for an existing database that was
created with MySQL Workbench.

First, I want to state that you need to understand that Django was designed
to help build simple, common use case web apps. Database skins, as someone
put it. The ORM is very easy to use, although it is not well prepared for
some things.

It also has a very nice advantage, which is being compatible with many
RDBMS so you can have postgresql on the production server and sqlite on
your development machine. This ends up being bad too. For instance, if it
is not possible to do something in a specific rdbms, for instance, let's
say blobs, Django will not be able to implement it because it has to
support those several rdbms.

There are many hacks ( http://djangosnippets.org/snippets/1597/ ) but hacks
will be hacks and you want your app to be reliable and not compromise your
code quality right from the start, right?

Many field options have equivalents in Django (primary key, auto
increment). You just have to find them. For auto increment there is
AutoField. Primary key is an option of the base Field (so you can provide
it as a keyword argument to any field).

Also if you want gis geometry data types (is that what you meant?) you can
install and use geodjango.

 I do not wish to use anything other than MySQL because of the complexity
of my storage needs (searchable text, PDF, Audio, Video and Photos). The
text searches will often be word phrase searches through thousands of
documents. I need the speed and the data type flexibility.

Are you sure mysql is up to it? For searchable text you will want to have
an index, and while mysql's FULLTEXT kind of works, it is clearly not a
core feature. Some options for it are enabled or disabled in compile time.
Also, I won't judge but it's often wrong to use a rdbms to store
PDF/audio/video. Those belong in the file system or nosql storage.

 How do I build or modify a model.py to do this. I really don't want to
write a model.py manually. That would really be a major pain. Workbench to
the database an import to Django and edit is my choice. Unfortunately, the
model.py produced has lost most of the functionality of the original
database and I can't seem to figure out how to fix it.

 Any help will be sincerely appreciated.

 Gary R.

A lot can be done by using the database manually. Django helps you a bit
here, for instance it gives you access to the database cursor for raw SQL
access to the database, supports custom fields, and every orm query chain
(like MyModel.objects.all().filter(...)) can be incremented with raw SQL
where and select clauses.

But you might not want to be using Django or at least not its ORM.
-- 
https://mail.python.org/mailman/listinfo/python-list

MySQL data types vs Django/Python data types

[Gary Roach]

Hi all,

System:
Debian Wheezy Linux
Python 2.7
Django 1.5
MySql 5.5

I am new to Python and Django and am having trouble matching Python data 
types with those of MySQL. MySQL has about 7 basic data types including 
Blobs, Binaries, etc. It also has a rich selection of geometry types. In 
addition, types like INT have 7 or 8 different options like Primary Key, 
zero fill, auto inc, etc. I can't seem to find anything in python to 
match these. I am trying to build a model.py for an existing database 
that was created with MySQL Workbench.


I do not wish to use anything other than MySQL because of the complexity 
of my storage needs (searchable text, PDF, Audio, Video and Photos). The 
text searches will often be word phrase searches through thousands of 
documents. I need the speed and the data type flexibility.


How do I build or modify a model.py to do this. I really don't want to 
write a model.py manually. That would really be a major pain. Workbench 
to the database an import to Django and edit is my choice. 
Unfortunately, the model.py produced has lost most of the functionality 
of the original database and I can't seem to figure out how to fix it.


Any help will be sincerely appreciated.

Gary R.
--
http://mail.python.org/mailman/listinfo/python-list

Re: save tuple of simple data types to disk (low memory foot print)

[Gelonida N]

On 10/29/2011 03:00 AM, Steven D'Aprano wrote:
 On Fri, 28 Oct 2011 22:47:42 +0200, Gelonida N wrote:
 
 Hi,

 I would like to save many dicts with a fixed amount of keys tuples to a
 file  in a memory efficient manner (no random, but only sequential
 access is required)

 
 What do you mean keys tuples?
Corrected phrase:
I would like to save many dicts with a fixed (and known) amount of keys
in a memory efficient manner (no random, but only sequential access is
required) to a file (which can later be sent over a slow expensive
network to other machines)

Example:
Every dict will have the keys 'timestamp', 'floatvalue', 'intvalue',
'message1', 'message2'
'timestamp' is an integer
'floatvalue' is a float
'intvalue' an int
'message1' is a string with a length of max 2000 characters, but can
often be very short
'message2' the same as message1

so a typical dict will look like
{ 'timetamp' : 12, 'floatvalue': 3.14159, 'intvalue': 42,
 'message1' : '', 'message2' : '=' * 1999 }



 What do you call many? Fifty? A thousand? A thousand million? How many
 items in each dict? Ten? A million?

File size can be between 100kb and over 100Mb per file. Files will be
accumulated over months.

I just want to use the smallest possible space, as the data is collected
over a certain time (days / months)  and will be transferred  via UMTS /
EDGE / GSM network, where the transfer takes already for quite small
data sets several minutes.

I want to reduce the transfer time, when requesting files on demand (and
the amount of data in order to not exceed the monthly quota)



 As the keys are the same for each entry  I considered converting them to
 tuples.
 
 I don't even understand what that means. You're going to convert the keys 
 to tuples? What will that accomplish?

 As the keys are the same for each entry  I considered converting them
(the before mentioned dicts) to tuples.

so the dict { 'timetamp' : 12, 'floatvalue': 3.14159, 'intvalue': 42,
 'message1' : '', 'message2' : '=' * 1999 }

would become
[ 12, 3.14159, 42, '', ''=' * 1999 ]
 
 
 The tuples contain only strings, ints (long ints) and floats (double)
 and the data types for each position within the tuple are fixed.

 The fastest and simplest way is to pickle the data or to use json. Both
 formats however are not that optimal.
 
 How big are your JSON files? 10KB? 10MB? 10GB?
 
 Have you tried using pickle's space-efficient binary format instead of 
 text format? Try using protocol=2 when you call pickle.Pickler.

No. This is probably already a big step forward.

As I know the data types if each element in the tuple I would however
prefer a representation, which is not storing the data types for each
typle over and over again (as they are the same for each dict / tuple)

 
 Or have you considered simply compressing the files?

Compression makes sense but the inital file format should be already
rather 'compact'

 
 I could store ints and floats with pack. As strings have variable length
 I'm not sure how to save them efficiently (except adding a length first
 and then the string.
 
 This isn't 1980 and you're very unlikely to be using 720KB floppies. 
 Premature optimization is the root of all evil. Keep in mind that when 
 you save a file to disk, even if it contains only a single bit of data, 
 the actual space used will be an entire block, which on modern hard 
 drives is very likely to be 4KB. Trying to compress files smaller than a 
 single block doesn't actually save you any space.

 
 
 Is there already some 'standard' way or standard library to store such
 data efficiently?
 
 Yes. Pickle and JSON plus zip or gzip.
 

pickle protocol-2 + gzip of the tuple derived from the dict, might be
good enough for the start.

I have to create a little more typical data in order to see how many
percent of my payload would consist of repeating the data types for each
tuple.




-- 
http://mail.python.org/mailman/listinfo/python-list

Re: save tuple of simple data types to disk (low memory foot print)

[Gelonida N]

On 10/29/2011 01:08 AM, Roy Smith wrote:
 In article mailman.2293.1319834877.27778.python-l...@python.org,
  Gelonida N gelon...@gmail.com wrote:
 
 I would like to save many dicts with a fixed amount of keys
 tuples to a file  in a memory efficient manner (no random, but only
 sequential access is required)
 
 There's two possible scenarios here.  One, which you seem to be 
 exploring, is to carefully study your data and figure out the best way 
 to externalize it which reduces volume.
 
 The other is to just write it out in whatever form is most convenient 
 (JSON is a reasonable thing to try first), and compress the output.  Let 
 the compression algorithms worry about extracting the entropy.  You may 
 be surprised at how well it works.  It's also an easy experiment to try, 
 so if it doesn't work well, at least it didn't cost you much to find out.


Yes I have to make some more tests to see the defference between
just compressing aplain format (JSON / pickle) and compressing the
'optimized' representation.




-- 
http://mail.python.org/mailman/listinfo/python-list

Re: save tuple of simple data types to disk (low memory foot print)

[Tim Chase]

On 10/29/11 11:44, Gelonida N wrote:

I would like to save many dicts with a fixed (and known) amount of keys
in a memory efficient manner (no random, but only sequential access is
required) to a file (which can later be sent over a slow expensive
network to other machines)

Example:
Every dict will have the keys 'timestamp', 'floatvalue', 'intvalue',
'message1', 'message2'
'timestamp' is an integer
'floatvalue' is a float
'intvalue' an int
'message1' is a string with a length of max 2000 characters, but can
often be very short
'message2' the same as message1

so a typical dict will look like
{ 'timetamp' : 12, 'floatvalue': 3.14159, 'intvalue': 42,
  'message1' : '', 'message2' : '=' * 1999 }




What do you call many? Fifty? A thousand? A thousand million? How many
items in each dict? Ten? A million?


File size can be between 100kb and over 100Mb per file. Files will be
accumulated over months.


If Steven's pickle-protocol2 solution doesn't quite do what you 
need, you can do something like the code below.  Gzip is pretty 
good at addressing...



Or have you considered simply compressing the files?

Compression makes sense but the inital file format should be
already rather 'compact'


...by compressing out a lot of the duplicate aspects.  Which also 
mitigates some of the verbosity of CSV.


It serializes the data to a gzipped CSV file then unserializes 
it.  Just point it at the appropriate data-source, adjust the 
column-names and data-types


-tkc

from gzip import GzipFile
from csv import writer, reader

data = [ # use your real data here
{
'timestamp': 12,
'floatvalue': 3.14159,
'intvalue': 42,
'message1': 'hello world',
'message2': '=' * 1999,
},
] * 1


f = GzipFile('data.gz', 'wb')
try:
w = writer(f)
for row in data:
w.writerow([
row[name] for name in (
# use your real col-names here
'timestamp',
'floatvalue',
'intvalue',
'message1',
'message2',
)])
finally:
f.close()

output = []
for row in reader(GzipFile('data.gz')):
d = dict((
(name, f(row[i]))
for i, (f,name) in enumerate((
# adjust for your column-names/data-types
(int, 'timestamp'),
(float, 'floatvalue'),
(int, 'intvalue'),
(str, 'message1'),
(str, 'message2'),

output.append(d)

# or

output = [
dict((
(name, f(row[i]))
for i, (f,name) in enumerate((
# adjust for your column-names/data-types
(int, 'timestamp'),
(float, 'floatvalue'),
(int, 'intvalue'),
(str, 'message1'),
(str, 'message2'),

for row in reader(GzipFile('data.gz'))
]
--
http://mail.python.org/mailman/listinfo/python-list

save tuple of simple data types to disk (low memory foot print)

[Gelonida N]

Hi,

I would like to save many dicts with a fixed amount of keys
tuples to a file  in a memory efficient manner (no random, but only
sequential access is required)

As the keys are the same for each entry  I considered converting them to
tuples.

The tuples contain only strings, ints (long ints) and floats (double)
and the data types for each position within the tuple are fixed.

The fastest and simplest way is to pickle the data or to use json.
Both formats however are not that optimal.


I could store ints and floats with pack. As strings have variable length
I'm not sure how to save them efficiently
(except adding a length first and then the string.

Is there already some 'standard' way or standard library to store
such data efficiently?

Thanks in advance for any suggestion.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: save tuple of simple data types to disk (low memory foot print)

[Roy Smith]

In article mailman.2293.1319834877.27778.python-l...@python.org,
 Gelonida N gelon...@gmail.com wrote:

 I would like to save many dicts with a fixed amount of keys
 tuples to a file  in a memory efficient manner (no random, but only
 sequential access is required)

There's two possible scenarios here.  One, which you seem to be 
exploring, is to carefully study your data and figure out the best way 
to externalize it which reduces volume.

The other is to just write it out in whatever form is most convenient 
(JSON is a reasonable thing to try first), and compress the output.  Let 
the compression algorithms worry about extracting the entropy.  You may 
be surprised at how well it works.  It's also an easy experiment to try, 
so if it doesn't work well, at least it didn't cost you much to find out.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: save tuple of simple data types to disk (low memory foot print)

[Steven D'Aprano]

On Fri, 28 Oct 2011 22:47:42 +0200, Gelonida N wrote:

 Hi,
 
 I would like to save many dicts with a fixed amount of keys tuples to a
 file  in a memory efficient manner (no random, but only sequential
 access is required)

What do you call many? Fifty? A thousand? A thousand million? How many 
items in each dict? Ten? A million?

What do you mean keys tuples?


 As the keys are the same for each entry  I considered converting them to
 tuples.

I don't even understand what that means. You're going to convert the keys 
to tuples? What will that accomplish?


 The tuples contain only strings, ints (long ints) and floats (double)
 and the data types for each position within the tuple are fixed.
 
 The fastest and simplest way is to pickle the data or to use json. Both
 formats however are not that optimal.

How big are your JSON files? 10KB? 10MB? 10GB?

Have you tried using pickle's space-efficient binary format instead of 
text format? Try using protocol=2 when you call pickle.Pickler.

Or have you considered simply compressing the files?


 I could store ints and floats with pack. As strings have variable length
 I'm not sure how to save them efficiently (except adding a length first
 and then the string.

This isn't 1980 and you're very unlikely to be using 720KB floppies. 
Premature optimization is the root of all evil. Keep in mind that when 
you save a file to disk, even if it contains only a single bit of data, 
the actual space used will be an entire block, which on modern hard 
drives is very likely to be 4KB. Trying to compress files smaller than a 
single block doesn't actually save you any space.


 Is there already some 'standard' way or standard library to store such
 data efficiently?

Yes. Pickle and JSON plus zip or gzip.


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

How to pretty-print ctypes composite data types?

[Grant Edwards]

I'm using ctypes with a library that requires a handful of structure
definitions.  The actual definitions and usage aren't a problem.  When
it comes time to print out the values in a structure or array, there
doesn't seem to be simple/general way to do that.  Am I missing
something?

I presume one can recursively iterate through the fields in a
structure and elements in an array, recursing on any composite types
and printing the values when one finds a type, but I'm surprised
that's not something that's already in the ctypes library somewhere --
the authors certainly seem to have thought of everything else.

-- 
Grant Edwards   grant.b.edwardsYow! I own seven-eighths of
  at   all the artists in downtown
  gmail.comBurbank!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python data types in c++ code

[Stefan Behnel]

Dan Stromberg, 07.03.2011 03:47:

On Sun, Mar 6, 2011 at 10:07 AM, Arthur Mc Coy wrote:

You know, they are still using SVN, they are
very loosely coupled to the past.


about SVN: I'm not sure it's really dying.


I hope it will.



Yes, a lot of distributed
development has moved off of SVN, and is better off for having done so, but
I believe some environments (especially corporate environments) just prefer
having a central server.


DVCSs do not prevent you from having a central server. They just keep you 
from having to rely on it. That's a big plus, also in corporate environments.


Worth a read: http://hginit.com/00.html

Stefan

--
http://mail.python.org/mailman/listinfo/python-list

c++ data types in python script

[Arthur Mc Coy]

Hi people,

I've used SWIG module to embed python inside c++ app. I pass a list of
objects (with lots of different properties of types string, float,
custom types like URL, Software and finally of list of strings).

Now I'm in python. URL and Software has str() method that converts
their value to string recognizable by JSON. But the problem is with
list of strings.

So, as I said I passed std::liststd::string myObjects to python
function.
Then I iterate it and for each object in myObjects I create a python
copy (serialize it) to be able to put into JSON format and store in
appropriate file.

object has property benchmarks of type liststring.
I do:
...
class PythonObject:
def __init__(self, object):
self.benchmarks = list()
for s in object.benchmarks:
self.benchmarks.append(s)
...
and it fails, also I do:
...
class PythonObject:
def __init__(self, object):
self.benchmarks = [unicode(s) for s in object.benchmarks]
...
and it fails, also I do:
...
class PythonObject:
def __init__(self, object):
for s in object.benchmarks:
print s[0] + s[1] + s[2]
print type(s)
...
and it fails printing
   wor
   type 'str'
   Segmentation fault (core dumped)
$
also I do:
...
class PythonObject:
def __init__(self, object):
self.benchmarks = unicode(object.benchmarks)
...
and it does not fail, instead it puts in JSON this string:
...
benchmarks: mymodule.StringList; proxy of Swig Object of type
'std::list std::string, std::allocator std::string   *' at
0xb63ed4e8,
...
but it is not what I need

What I'm trying to stress is that c++ objects should be converted
(serialized) before putting them into json. Otherwise type errors
occur and process fails.


I love learning python and hope somebody may suggest me or tell
something.

Thank you all anyway!

Arthur
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: c++ data types in python script

[Arthur Mc Coy]

Great!

The solution is to use self.benchmarks = list(object.benchmarks).

Now I'm battling with time_t type. C++ time_t converts to python int
but it causes memore leaks due to destructor absence. I'm trying to
figure it out. If anyone know, please share your thoughts.

Be happy :)

Arthur
-- 
http://mail.python.org/mailman/listinfo/python-list

python data types in c++ code

[Arthur Mc Coy]

Hi all,


Ok, I managed to work with c++ data types in python and can store
serialize c++ objects to store in json.

Now the task is backward. I wrote a c++ code to get the list of
objects using again python interface. The list of objects is returned.
PyList.

But I can't see how to convert PyObject as element of PyList to c++
data type or even how to convert or iterate PyList which is PyObject
too.


Please, whisper me the proper way I will go for.
Be happy :)

Arthur
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python data types in c++ code

[Stefan Behnel]

Arthur Mc Coy, 06.03.2011 17:40:

Ok, I managed to work with c++ data types in python and can store
serialize c++ objects to store in json.

Now the task is backward. I wrote a c++ code to get the list of
objects using again python interface. The list of objects is returned.
PyList.

But I can't see how to convert PyObject as element of PyList to c++
data type or even how to convert or iterate PyList which is PyObject
too.

Please, whisper me the proper way I will go for.


You mentioned using SWIG, but I'd actually suggest using Cython instead. It 
will make it a lot easier (and faster) for you to convert data types 
between Python and C/C++ (and will also allow you to skip over most 
ref-counting problems). Since you appear to know C++ anyway, you should be 
able to get comfortable with it quite quickly.


http://cython.org

Stefan

--
http://mail.python.org/mailman/listinfo/python-list

Re: python data types in c++ code

[Arthur Mc Coy]

Ok people,


I do:
for (Py_ssize_t i = 0; i PyList_Size(py_list); ++i) {
 PuObject* obj = PyList_GetItem(py_list, i);
 if (obj != NULL) {
   // howto get obj properties of different types
(map, list, string, int, bool) ?
 }
}

Very clear code. The question is in the comment :)
Let's play more,
it's like jazz,
let's explore,
Baby baby baby bass


Keep yourselves,
Arthur
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python data types in c++ code

[Arthur Mc Coy]

Stefan, great suggestion !!

I will definitely bookmark this page and consider later. But my
project dictate me use SWIG. I'm almost near the happy subend. Very
curios on question I asked in previous message.

Dear all, I love you too much :)

Arthur
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python data types in c++ code

[Arthur Mc Coy]

Stephan, you are lead developer over there :))) It's marketing,
however very nice solution.. I will propose my bosses to rediscover
their world assumptions. You know, they are still using SVN, they are
very loosely coupled to the past. I have to wash their brains...

Meantime still battling with PyObject properties which are so diverse
and disperse that I don't know...


Keep talking guys


Arthur
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python data types in c++ code

[Arthur Mc Coy]

In short, the answer is to use PyObject dictionary ?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python data types in c++ code

[Stefan Behnel]

Arthur Mc Coy, 06.03.2011 19:07:

Stephan, you are lead developer over there :))) It's marketing,


Let's say, as a core developer of Cython, I'm well aware of it's virtues, 
and I can tell you that my suggestion is actually well backed by the user 
feedback we get. You will find some of it on the right side of cython.org. 
The selection there is obviously biased, but I can confirm that it really 
matches with most of the non-technical feedback we get. Cython has a lot of 
happy users, and that's very rewarding to its core developers.


Stefan

--
http://mail.python.org/mailman/listinfo/python-list

Re: python data types in c++ code

[Miki Tebeka]

// howto get obj properties of different types
 (map, list, string, int, bool) ?
Python's C API is very well documented. You can use 
http://docs.python.org/c-api/object.html?highlight=pyobject#PyObject_GetAttr to 
get attributes or 
http://docs.python.org/c-api/object.html?highlight=pyobject#PyObject_GetItem to 
get items (from dictionaries and maps).

HTH
--
Miki Tebeka miki.teb...@gmail.com
http://pythonwise.blogspot.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python data types in c++ code

[Dan Stromberg]

On Sun, Mar 6, 2011 at 10:07 AM, Arthur Mc Coy 1984docmc...@gmail.comwrote:

 You know, they are still using SVN, they are
 very loosely coupled to the past.


Cython's very nice if you don't plan to do more than C/C++ with Python.
SWIG might be better if you intend to do more VHLL's than Python alone.

But about SVN: I'm not sure it's really dying.  Yes, a lot of distributed
development has moved off of SVN, and is better off for having done so, but
I believe some environments (especially corporate environments) just prefer
having a central server.
-- 
http://mail.python.org/mailman/listinfo/python-list

[issue6302] email.header.decode_header data types are inconsistent and incorrectly documented

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Drat, missed this one when I was reviewing my issues for feature requests 
because I didn't change the type :(

--
versions: +Python 3.3 -Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6302
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6302] email.header.decode_header data types are inconsistent and incorrectly documented

[R. David Murray]

Changes by R. David Murray rdmur...@bitdance.com:


--
nosy:  -BreamoreBoy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6302
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6302] email.header.decode_header data types are inconsistent and incorrectly documented

[Barry A. Warsaw]

Barry A. Warsaw ba...@python.org added the comment:

In email6, can we at least make tuple returning methods return namedtuples 
instead?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6302
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6302] email.header.decode_header data types are inconsistent and incorrectly documented

[Barry A. Warsaw]

Barry A. Warsaw ba...@python.org added the comment:

I agree that it makes sense to have consistent types in the output.  As for 
whether to add a new method or fix the existing one, I'm a bit torn, but I'd 
probably opt for fixing the existing function rather than adding a new one, 
just because I think there are few Python 3 applications out there that are 
counting on the old behavior.  (I could be wrong though ;).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6302
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6302] email.header.decode_header data types are inconsistent and incorrectly documented

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 I agree that it makes sense to have consistent types in the output.
 As for whether to add a new method or fix the existing one, I'm a bit
 torn, but I'd probably opt for fixing the existing function rather
 than adding a new one, just because I think there are few Python 3
 applications out there that are counting on the old behavior.  (I
 could be wrong though ;).

The point of a new method is to return the header as a human-readable
string, rather than a list of tuples. It has added value besides leaving
the old method alone ;)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6302
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6302] email.header.decode_header data types are inconsistent and incorrectly documented

[Barry A. Warsaw]

Barry A. Warsaw ba...@python.org added the comment:

The point of a new method is to return the header as a human-readable
string, rather than a list of tuples. It has added value besides
leaving the old method alone ;)

+1 then! :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6302
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6302] email.header.decode_header data types are inconsistent and incorrectly documented

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 I think nttplib's use case can be satisfied via the issue 4661 patch
 coupled with the decode_header bytes-recovery enhancement.

I don't really understand how that could.
nntplib needs to decode (in the decode_header sense) headers containing 
unescaped as well as escaped non-ASCII chars. Encoding with ascii clearly 
breaks the first use case.

Since the decode_header() API is so silly to begin with, I'd suggest providing 
another higher-level API instead. One which takes an str and returns another 
str (not bytes!) instead of that list of tuples.

If you really want to fix the current decode_header(), I'd recommend adding 
an optional encoding parameter so that nntplib can give utf-8 instead of 
ascii. nntplib and other consumers will still have to decode back in order to 
get an str, which makes the whole encoding thing a bit useless.

Oh, and instead of None, it would be nicer to give the actual encoding (e.g. 
ascii). It's no fun trying to guess.

--
nosy: +barry

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6302
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6302] email.header.decode_header data types are inconsistent and incorrectly documented

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Yes, that was a late night post and as I was falling asleep I realized that I 
was wrong.

Certainly decode_header_as_string is a function most people using the email 
package will want and will re-implement in one form or another, so I think it 
is a good idea to add it.  I will take a look at the patch later.  And with 
such a function added we can leave decode_header alone for backward 
compatibility.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6302
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6302] email.header.decode_header data types are inconsistent and incorrectly documented

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Here is a patch that makes the output consistently (bytes, string) pairs.  This 
is definitely a potential backward compatibility issue, but in general code 
which compensates for the old behavior should work fine with the new behavior, 
since it was always possible to get a (bytes, None) tuple back as a result, so 
most code should be handling that case correctly already.

IMO this change is nevertheless worthwhile; especially since if the patch in 
issue 4661 is accepted decode_header can be enhanced so that it will provide a 
way to obtain the bytes version of a header containing (RFC invalid) non-ASCII 
bytes.

Note that this breaks one of the tests in nttplib, so backward compatibility 
really is an issue, unfortunately.  I think nttplib's use case can be satisfied 
via the issue 4661 patch coupled with the decode_header bytes-recovery 
enhancement.

--
dependencies: +email.parser: impossible to read messages encoded in a different 
encoding
keywords: +patch
nosy: +pitrou
Added file: http://bugs.python.org/file19115/decode_header.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6302
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6302] email.header.decode_header data types are inconsistent and incorrectly documented

[R. David Murray]

Changes by R. David Murray rdmur...@bitdance.com:


--
versions:  -Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6302
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6302] email.header.decode_header data types are inconsistent and incorrectly documented

[Mark Lawrence]

Mark Lawrence breamore...@yahoo.co.uk added the comment:

Anyone got any comments to make on this?  Should 2.7 also be included?

--
nosy: +BreamoreBoy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6302
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6302] email.header.decode_header data types are inconsistent and incorrectly documented

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

No, this is a 3.x only problem.  And my main comment is that decode_headers 
ought to go away as an API :)

But I'll try to fix the inconsistent data types problem before 3.2.

--
assignee:  - r.david.murray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6302
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Pyserial and pysqlite data types, need advice

[jon vs. python]

The best thing I've found is this:

http://eli.thegreenplace.net/2009/05/29/storing-blobs-in-a-sqlite-db-with-pythonpysqlite/

On Mon, May 17, 2010 at 5:05 PM, jon vs. python jonvspyt...@gmail.comwrote:

 Hi,
 I'm trying to store frames received via serial port (using Pyserial) into a
 sqlite database (using Pysqlite) in order to perform off-line processing.
 Thus I could use both SQL's power and Python's magic to make everything
 easier. I'd like my code to be generic and work both for binary and ascii
 protocols, too.

 Which kind of data should I use to store the frames?
 Should I store every byte as a char in VARCHAR? (This seems to fail when
 trying to store non printable characters).
 Should I encapsulate frames in buffer objects and store them in BLOBs?
 (This seems to work but hides content and thus doesn't allow to use database
 operations directly on the data)
 I've also tried, unsuccessfully, to use bytearrays with pysqlite...

 Any suggestion?

 Thanks, Jon.

-- 
http://mail.python.org/mailman/listinfo/python-list

Pyserial and pysqlite data types, need advice

[jon vs. python]

Hi,
I'm trying to store frames received via serial port (using Pyserial) into a
sqlite database (using Pysqlite) in order to perform off-line processing.
Thus I could use both SQL's power and Python's magic to make everything
easier. I'd like my code to be generic and work both for binary and ascii
protocols, too.

Which kind of data should I use to store the frames?
Should I store every byte as a char in VARCHAR? (This seems to fail when
trying to store non printable characters).
Should I encapsulate frames in buffer objects and store them in BLOBs? (This
seems to work but hides content and thus doesn't allow to use database
operations directly on the data)
I've also tried, unsuccessfully, to use bytearrays with pysqlite...

Any suggestion?

Thanks, Jon.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Feeding differeent data types to a class instance?

[Christian]

On Mar 14, 2:16 pm, kuru maymunbe...@gmail.com wrote:
 Hi

 Thank you so much for all these great suggestions. I will have time
 today to try all these and see which one works best for me

Answers to your question have spread onto the Python blogosphere as
well.

http://pythonconquerstheuniverse.wordpress.com/2010/03/17/multiple-constructors-in-a-python-class/

http://blog.dowski.com/2010/03/17/my-take-on-multiple-constructors/

That second one is my response.

Christian


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Feeding differeent data types to a class instance?

[Steve Holden]

kuru wrote:
 Hi
 
 I have couple classes in the form of
 
 class Vector:
   def __init__(self,x,y,z):
self.x=x
self.y=y
self.z=z
 
 This works fine for me. However I want to be able to provide a list,
 tuple as well as individual arguments like below
 
 myvec=Vector(1,2,3)
 
 This works well
 
 
 However I  also want to be able to do
 
 vect=[1,2,3]
 
 myvec=Vec(vect)
 
 I want this class to accept multiple data types but process them as
 they are same when the classs deals with the instances.
 
 thanks
 
 
With your existing class you can use Python's ability to transform a
list or tuple into individual arguments using the * notation:

 class Vector:
...   def __init__(self,x,y,z):
...self.x=x
...self.y=y
...self.z=z
...
 vl = [1, 2, 3]
 v = Vector(*vl)
 v.x
1
 v.y
2
 v.z
3


Will this do? It seems much simpler than rewriting an already
satisfactory class.

regards
 Steve
-- 
Steve Holden   +1 571 484 6266   +1 800 494 3119
See PyCon Talks from Atlanta 2010  http://pycon.blip.tv/
Holden Web LLC http://www.holdenweb.com/
UPCOMING EVENTS:http://holdenweb.eventbrite.com/

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Feeding differeent data types to a class instance?

[kuru]

Hi

Thank you so much for all these great suggestions. I will have time
today to try all these and see which one works best for me

cheers
-- 
http://mail.python.org/mailman/listinfo/python-list

Feeding differeent data types to a class instance?

[kuru]

Hi

I have couple classes in the form of

class Vector:
  def __init__(self,x,y,z):
   self.x=x
   self.y=y
   self.z=z

This works fine for me. However I want to be able to provide a list,
tuple as well as individual arguments like below

myvec=Vector(1,2,3)

This works well


However I  also want to be able to do

vect=[1,2,3]

myvec=Vec(vect)

I want this class to accept multiple data types but process them as
they are same when the classs deals with the instances.

thanks


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Feeding differeent data types to a class instance?

[Rhodri James]

On Sun, 14 Mar 2010 00:34:55 -, kuru maymunbe...@gmail.com wrote:


Hi

I have couple classes in the form of

class Vector:
  def __init__(self,x,y,z):
   self.x=x
   self.y=y
   self.z=z

This works fine for me. However I want to be able to provide a list,
tuple as well as individual arguments like below

myvec=Vector(1,2,3)

This works well


However I  also want to be able to do

vect=[1,2,3]

myvec=Vec(vect)


You can do something like:

class Vector(object):
  def __init__(self, x, y=None, z=None):
if isinstance(x, list):
  self.x = x[0]
  self.y = x[1]
  self.z = x[2]
else:
  self.x = x
  self.y = y
  self.z = z

but this gets messy quite quickly.  The usual wisdom these days is to  
write yourself a separate class method to create your object from a  
different type:


class Vector(object):
  ... def __init__ as you did before ...

  @classmethod
  def from_list(cls, lst):
return cls(lst[0], lst[1], lst[2])

vect = [1,2,3]
myvec = Vector.from_list(vect)


--
Rhodri James *-* Wildebeeste Herder to the Masses
--
http://mail.python.org/mailman/listinfo/python-list

Re: Feeding differeent data types to a class instance?

[Steven D'Aprano]

On Sat, 13 Mar 2010 16:34:55 -0800, kuru wrote:

 I want this class to accept multiple data types but process them as they
 are same when the classs deals with the instances.

The usual term for this is polymorphism.

 myvec=Vector(1,2,3)

 vect=[1,2,3]
 myvec=Vec(vect)

I assume you mean Vector in the last line.

I find this the easiest way to handle this situation:

class Vector(object, *args):
if len(args) == 1:
# Assume the caller passed a list argument.
args = args[0]
x, y, z = args  # Unpack the arguments.
# then process as normal.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

[issue6302] email.header.decode_header data types are inconsistent and incorrectly documented

[R. David Murray]

New submission from R. David Murray rdmur...@bitdance.com:

decode_header only accepts str as input.  If the input contains no
encoded words, the output is str (ie: the input string) and None.  If it
does contain encoded words, the output is pairs of bytes plus str
charset identifiers (or None).  This makes it difficult to use this
function to decode headers, since one has to test for the special case
of str output when there are no encoded words.

I think decode_header should take bytes as input, and output (bytes.
str) pairs consistently.

In any case, the documentation is wrong since it says it returns
strings.  The example is also wrong, since the actual output is:

[(b'p\xf6stal', 'iso-8859-1')]

--
components: Library (Lib)
messages: 89488
nosy: r.david.murray
priority: normal
severity: normal
stage: test needed
status: open
title: email.header.decode_header data types are inconsistent and incorrectly 
documented
type: behavior
versions: Python 3.1, Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6302
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Byte oriented data types in python

[Ravi]

 Take a look at the struct and ctypes modules.

struct is really not the choice. it returns an expanded string of the
data and this means larger latency over bluetooth.

ctypes is basically for the interface with libraries written in C
(this I read from the python docs)

--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Ravi]

On Jan 25, 12:52 am, Martin v. Löwis mar...@v.loewis.de wrote:
  packet_type (1 byte unsigned) || packet_length (1 byte unsigned) ||
  packet_data(variable)

  How to construct these using python data types, as int and float have
  no limits and their sizes are not well defined.

 In Python 2.x, use the regular string type: chr(n) will create a single
 byte, and the + operator will do the concatenation.

 In Python 3.x, use the bytes type (bytes() instead of chr()).

This looks really helpful thanks!

--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Steve Holden]

Ravi wrote:
 Take a look at the struct and ctypes modules.
 
 struct is really not the choice. it returns an expanded string of the
 data and this means larger latency over bluetooth.
 
If you read the module documentation more carefully you will see that it
converts between the various native data types and character strings.
Thus each native data type occupies only as many bytes as are required
to store it in its native form (modulo any alignments needed).

 ctypes is basically for the interface with libraries written in C
 (this I read from the python docs)
 
I believe it *is* the struct module you need.

regards
 Steve
-- 
Steve Holden+1 571 484 6266   +1 800 494 3119
Holden Web LLC  http://www.holdenweb.com/

--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Grant Edwards]

On 2009-01-25, Ravi ra.ravi@gmail.com wrote:

 Take a look at the struct and ctypes modules.

 struct is really not the choice. it returns an expanded string of the
 data and this means larger latency over bluetooth.

I don't know what you mean by returns an expanded string of
the data.

I do know that struct does exactly what you requested.

It converts between Python objects and what is bascially a C
struct where you specify the endianness of each field and
what sort of packing/padding you want.

I use the struct module frequenty to impliment binary,
communications protocols in Python.  I've used Python/struct
with transport layers ranging from Ethernet (raw, TCP, and UDP)
to async serial, to CAN.

-- 
--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Stephen Hansen]

On Sun, Jan 25, 2009 at 7:27 AM, Ravi ra.ravi@gmail.com wrote:


  Take a look at the struct and ctypes modules.

 struct is really not the choice. it returns an expanded string of the
 data and this means larger latency over bluetooth.


Noo... struct really IS the choice; that is the explicit purpose of the
struct library. I went and included an example too which you're not noticing
:)

Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type help, copyright, credits or license for more information.
 import struct
 data = struct.pack(BB4s, 1, 4, this)
 data
'\x01\x04this'


That's precisely six raw bytes which is exactly what you specified you
needed: exactly one unsigned byte for packet type, one unsigned byte for
length, and four bytes for the data after. In real life you'd probably use
something besides 4s to pack what you marked as other and variable, of
course, but still. Replace it with whatever your message requires.

Really, this is how you do line protocols ;) Well there's other ways, but...

--S
--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Martin v. Löwis]

 Take a look at the struct and ctypes modules.
 struct is really not the choice. it returns an expanded string of the
 data and this means larger latency over bluetooth.
 
 I don't know what you mean by returns an expanded string of
 the data.
 
 I do know that struct does exactly what you requested.

I disagree. He has a format (type, length, value), with the
value being variable-sized. How do you do that in the struct
module?

 It converts between Python objects and what is bascially a C
 struct where you specify the endianness of each field and
 what sort of packing/padding you want.

Sure. However, in the specific case, there is really no C
struct that can reasonably represent the data. Hence you
cannot really use the struct module.

 I use the struct module frequenty to impliment binary,
 communications protocols in Python.  I've used Python/struct
 with transport layers ranging from Ethernet (raw, TCP, and UDP)
 to async serial, to CAN.

Do you use it for the fixed-size parts, or also for the variable-sized
data?

Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Grant Edwards]

On 2009-01-25, Martin v. Löwis mar...@v.loewis.de wrote:

 Take a look at the struct and ctypes modules.
 struct is really not the choice. it returns an expanded string of the
 data and this means larger latency over bluetooth.
 
 I don't know what you mean by returns an expanded string of
 the data.
 
 I do know that struct does exactly what you requested.

 I disagree. He has a format (type, length, value), with the
 value being variable-sized. How do you do that in the struct
 module?

You construct a format string for the value portion based on
the type/length header.

 It converts between Python objects and what is bascially a C
 struct where you specify the endianness of each field and
 what sort of packing/padding you want.

 Sure. However, in the specific case, there is really no C
 struct that can reasonably represent the data.

I don't see how that can be the case.  There may not be a
single C struct that can represent all frames, but for every
frame you should be able to come up with a C struct that can
represent that frame.

 Hence you cannot really use the struct module.

Perhaps I don't understand his requirements, but I use the
struct module for protocols with type/len/value sorts of
packets.

 I use the struct module frequenty to impliment binary,
 communications protocols in Python.  I've used Python/struct
 with transport layers ranging from Ethernet (raw, TCP, and
 UDP) to async serial, to CAN.

 Do you use it for the fixed-size parts, or also for the
 variable-sized data?

Both.  For varible size/format stuff you decode the first few
bytes and use them to figure out what format/layout to use for
the next chunk of data.  It's pretty much the same thing you do
in other languages.

-- 
Grant

--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Martin v. Löwis]

 I disagree. He has a format (type, length, value), with the
 value being variable-sized. How do you do that in the struct
 module?
 
 You construct a format string for the value portion based on
 the type/length header.

Can you kindly provide example code on how to do this?

 I don't see how that can be the case.  There may not be a
 single C struct that can represent all frames, but for every
 frame you should be able to come up with a C struct that can
 represent that frame.

Sure. You would normally have a struct such as

struct TLV{
  char type;
  char length;
  char *data;
};

However, the in-memory representation of that struct is *not*
meant to be sent over the wire. In particular, the character
pointer has no meaning outside the address space, and is thus
not to be sent.

 Both.  For varible size/format stuff you decode the first few
 bytes and use them to figure out what format/layout to use for
 the next chunk of data.  It's pretty much the same thing you do
 in other languages.

In the example he gave, I would just avoid using the struct module
entirely, as it does not provide any additional value:

def encode(type, length, value):
  return chr(type)+chr(length)+value

Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[John Machin]

On Jan 26, 2:28 am, Ravi ra.ravi@gmail.com wrote:
 On Jan 25, 12:52 am, Martin v. Löwis mar...@v.loewis.de wrote:

   packet_type (1 byte unsigned) || packet_length (1 byte unsigned) ||
   packet_data(variable)

   How to construct these using python data types, as int and float have
   no limits and their sizes are not well defined.

  In Python 2.x, use the regular string type: chr(n) will create a single
  byte, and the + operator will do the concatenation.

  In Python 3.x, use the bytes type (bytes() instead of chr()).

 This looks really helpful thanks!

Provided that you don't take Martin's last sentence too literally :-)


| Python 2.6.1 (r261:67517, Dec  4 2008, 16:51:00) [MSC v.1500 32 bit
(Intel)] on win32
|  p_data = babcd # Omit the b prefix if using 2.5 or earlier
|  p_len = len(p_data)
|  p_type = 3
|  chr(p_type) + chr(p_len) + p_data
| '\x03\x04abcd'

| Python 3.0 (r30:67507, Dec  3 2008, 20:14:27) [MSC v.1500 32 bit
(Intel)] on win32
|  p_data = babcd
|  p_len = len(p_data)
|  p_type = 3
|  bytes(p_type) + bytes(p_len) + p_data # literal translation
| b'\x00\x00\x00\x00\x00\x00\x00abcd'
|  bytes(3)
| b'\x00\x00\x00'
|  bytes(10)
| b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
|  bytes([p_type]) + bytes([p_len]) + p_data
| b'\x03\x04abcd'
|  bytes([p_type, p_len]) + p_data
| b'\x03\x04abcd'

Am I missing a better way to translate chr(n) from 2.x to 3.x? The
meaning assigned to bytes(n) in 3.X is interesting:

2.X:
nuls = '\0' * n
out_byte = chr(n)

3.X:
nuls = b'\0' * n
or
nuls = bytes(n)
out_byte = bytes([n])

Looks to me like there was already a reasonable way of getting a bytes
object containing a variable number of zero bytes. Any particular
reason why bytes(n) was given this specialised meaning? Can't be the
speed, because the speed of bytes(n) on my box is about 50% of the
speed of the * expression for n = 16 and about 65% for n = 1024.

Cheers,
John
--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Martin v. Löwis]

 Looks to me like there was already a reasonable way of getting a bytes
 object containing a variable number of zero bytes. Any particular
 reason why bytes(n) was given this specialised meaning?

I think it was because bytes() was originally mutable, and you need a
way to create a buffer of n bytes. Now that bytes() ended up immutable
(and bytearray was added), it's perhaps not so useful anymore. Of
course, it would be confusing if bytes(4) created a sequence of one
byte, yet bytearray(4) created four bytes.

Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Grant Edwards]

On 2009-01-25, Martin v. Löwis mar...@v.loewis.de wrote:

 You construct a format string for the value portion based on
 the type/length header.

 Can you kindly provide example code on how to do this?

OK, something like this to handle received data where there is
an initial 8-bit type field that is 1 for 16-bit unsigned
integers in network byte-order, 2 for 32-bit IEEE floats in
network byte-order.  We'll further assume that the 'length'
field comes next as a 16 bit unsigned value in network order
and represents how many objects of the specified type follow:

  dtype = ord(rawdata[0])
  dcount = struct.unpack(!H,rawdata[1:3])
  if dtype == 1:
 fmtstr = ! + H*dcount
  elif dtype == 2:
 fmtstr = ! + f*dcount
  rlen = struct.calcsize(fmtstr)
  
  data = struct.unpack(fmtstr,rawdata[3:3+rlen])
  
  leftover = rawdata[3+rlen:]

 I don't see how that can be the case.  There may not be a
 single C struct that can represent all frames, but for every
 frame you should be able to come up with a C struct that can
 represent that frame.

 Sure. You would normally have a struct such as

 struct TLV{
   char type;
   char length;
   char *data;
 };

 However, the in-memory representation of that struct is *not*
 meant to be sent over the wire. In particular, the character
 pointer has no meaning outside the address space, and is thus
 not to be sent.

Well if it's not representing the layout of the data we're
trying to deal with, then it's irrelevent.  We are talking
about how convert python objects to/from data in the
'on-the-wire' format, right?

Or isn't that what the OP is asking about?

 Both.  For varible size/format stuff you decode the first few
 bytes and use them to figure out what format/layout to use for
 the next chunk of data.  It's pretty much the same thing you do
 in other languages.

 In the example he gave, I would just avoid using the struct module
 entirely, as it does not provide any additional value:

 def encode(type, length, value):
   return chr(type)+chr(length)+value

Like this?

 def encode(type,length,value):
...  return chr(type)+chr(length)+value
... 
 print encode('float', 1, 3.14159)
Traceback (most recent call last):
  File stdin, line 1, in module
  File stdin, line 2, in encode
TypeError: an integer is required
 

-- 
Grant

--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Martin v. Löwis]

   dtype = ord(rawdata[0])
   dcount = struct.unpack(!H,rawdata[1:3])
   if dtype == 1:
  fmtstr = ! + H*dcount
   elif dtype == 2:
  fmtstr = ! + f*dcount
   rlen = struct.calcsize(fmtstr)
   
   data = struct.unpack(fmtstr,rawdata[3:3+rlen])
   
   leftover = rawdata[3+rlen:]

Unfortunately, that does not work in the example. We have
a message type (an integer), and a variable-length string.
So how do you compute the struct format for that?

 Sure. You would normally have a struct such as

 struct TLV{
   char type;
   char length;
   char *data;
 };

 However, the in-memory representation of that struct is *not*
 meant to be sent over the wire. In particular, the character
 pointer has no meaning outside the address space, and is thus
 not to be sent.
 
 Well if it's not representing the layout of the data we're
 trying to deal with, then it's irrelevent.  We are talking
 about how convert python objects to/from data in the
 'on-the-wire' format, right?

Right: ON-THE-WIRE, not IN MEMORY. In memory, there is a
pointer. On the wire, there are no pointers.

 Like this?
 
 def encode(type,length,value):
 ...  return chr(type)+chr(length)+value
 ... 
 print encode('float', 1, 3.14159)
 Traceback (most recent call last):
   File stdin, line 1, in module
   File stdin, line 2, in encode
 TypeError: an integer is required

No:

py CONNECT_REQUEST=17
py payload=call me
py encode(CONNECT_REQUEST, len(payload), payload)
'\x11\x07call me'

Regards,
Martin

--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Grant Edwards]

On 2009-01-25, Martin v. Löwis mar...@v.loewis.de wrote:
   dtype = ord(rawdata[0])
   dcount = struct.unpack(!H,rawdata[1:3])
   if dtype == 1:
  fmtstr = ! + H*dcount
   elif dtype == 2:
  fmtstr = ! + f*dcount
   rlen = struct.calcsize(fmtstr)
   
   data = struct.unpack(fmtstr,rawdata[3:3+rlen])
   
   leftover = rawdata[3+rlen:]

 Unfortunately, that does not work in the example. We have
 a message type (an integer), and a variable-length string.
 So how do you compute the struct format for that?

I'm confused. Are you asking for an introductory tutorial on
programming in Python?

 Right: ON-THE-WIRE, not IN MEMORY. In memory, there is a
 pointer. On the wire, there are no pointers.

I don't understand your point.

 py CONNECT_REQUEST=17
 py payload=call me
 py encode(CONNECT_REQUEST, len(payload), payload)
 '\x11\x07call me'

If all your data is comprised of 8-bit bytes, then you don't
need the struct module.

-- 
Grant
--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Martin v. Löwis]

 Unfortunately, that does not work in the example. We have
 a message type (an integer), and a variable-length string.
 So how do you compute the struct format for that?
 
 I'm confused. Are you asking for an introductory tutorial on
 programming in Python?

Perhaps. I honestly do not know how to deal with variable-sized
strings in the struct module in a reasonable way, and thus believe
that this module is incapable of actually supporting them
(unless you use inappropriate trickery).

However, as you keep claiming that the struct module is what
should be used, I must be missing something about the struct
module.

 I don't understand your point.
 
 py CONNECT_REQUEST=17
 py payload=call me
 py encode(CONNECT_REQUEST, len(payload), payload)
 '\x11\x07call me'
 
 If all your data is comprised of 8-bit bytes, then you don't
 need the struct module.

Go back to the original message of the OP. It says

#  I have following packet format which I have to send over Bluetooth.
# packet_type (1 byte unsigned) || packet_length (1 byte unsigned) ||
# packet_data(variable)

So yes, all his date is comprised of 8-bit bytes, and yes, he doesn't
need the struct module. Hence I'm puzzled why people suggest that
he uses the struct module.

I think the key answer is use the string type, it is appropriate
to represent byte oriented data in python (also see the subject
of this thread)

Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Grant Edwards]

On 2009-01-25, Martin v. Löwis mar...@v.loewis.de wrote:
 Unfortunately, that does not work in the example. We have
 a message type (an integer), and a variable-length string.
 So how do you compute the struct format for that?
 
 I'm confused. Are you asking for an introductory tutorial on
 programming in Python?

 Perhaps. I honestly do not know how to deal with variable-sized
 strings in the struct module in a reasonable way, and thus believe
 that this module is incapable of actually supporting them
 (unless you use inappropriate trickery).

It deals with variable sized fields just fine:

dtype = 18
dlength = 32
format = !BB%ds % dlength

rawdata = struct.pack(format, (dtype,dlength,data))

 However, as you keep claiming that the struct module is what
 should be used, I must be missing something about the struct
 module.

http://docs.python.org/library/struct.html

 I don't understand your point.
 
 py CONNECT_REQUEST=17
 py payload=call me
 py encode(CONNECT_REQUEST, len(payload), payload)
 '\x11\x07call me'
 
 If all your data is comprised of 8-bit bytes, then you don't
 need the struct module.

 Go back to the original message of the OP. It says

 #  I have following packet format which I have to send over Bluetooth.
 # packet_type (1 byte unsigned) || packet_length (1 byte unsigned) ||
 # packet_data(variable)

 So yes, all his date is comprised of 8-bit bytes,

He doesn't specify what format the packet_data is, and we all
assumed he needed to handle conversion of various data types
to/from raw byte-strings.

 and yes, he doesn't need the struct module. Hence I'm puzzled
 why people suggest that he uses the struct module.

We all assumed that packet_data might contain values of
various types such as 16 or 32 bit integers, floating point
values -- that packet_data was not solely arbitrary-length
strings of 8-bit bytes.

 I think the key answer is use the string type, it is
 appropriate to represent byte oriented data in python (also
 see the subject of this thread)

I, for one, interpreted byte-oriented to mean that the data
was received/sent as blocks of bytes but needed to be converted
into other data types.  If the data really is just strings of
bytes, and it's sent as strings of bytes, then I have no idea
what the OP was asking, since there's nothing that needs to be
done with the data.

-- 
Grant

--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Martin v. Löwis]

 It deals with variable sized fields just fine:
 
 dtype = 18
 dlength = 32
 format = !BB%ds % dlength
 
 rawdata = struct.pack(format, (dtype,dlength,data))

I wouldn't call this just fine, though - it involves
a % operator to even compute the format string. IMO,
it is *much* better not to use the struct module for this
kind of problem, and instead rely on regular string
concatenation.

Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Grant Edwards]

On 2009-01-25, Martin v. Löwis mar...@v.loewis.de wrote:
 It deals with variable sized fields just fine:
 
 dtype = 18
 dlength = 32
 format = !BB%ds % dlength
 
 rawdata = struct.pack(format, (dtype,dlength,data))

 I wouldn't call this just fine, though - it involves
 a % operator to even compute the format string. IMO,
 it is *much* better not to use the struct module for this
 kind of problem, and instead rely on regular string
 concatenation.

If all you need to do is concatenate strings, then you're
correct, there's no advantage to using struct or ctypes.

If you need a generic way to deal with arbitrary data types,
then that's what the struct and ctypes modules are designed to
do.  The protocols I've implemented always required the ability
to deal with integers greater than 8 bits wide as well as
various other data types.

-- 
Grant

--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Stephen Hansen]

 However, as you keep claiming that the struct module is what
 should be used, I must be missing something about the struct
 module.


You seem to be focusing overly on the C Struct part of the description of
what the struct module does, instead of the part where it says, packed
binary data and This can be used in handling binary data stored in files
or
from network connections

The struct module is very useful when generating or interpreting a stream
of binary data which has a set structured format, in bytes and with set
kinds
of data types.



 #  I have following packet format which I have to send over Bluetooth.
 # packet_type (1 byte unsigned) || packet_length (1 byte unsigned) ||
 # packet_data(variable)

 So yes, all his date is comprised of 8-bit bytes, and yes, he doesn't
 need the struct module. Hence I'm puzzled why people suggest that
 he uses the struct module.

 I think the key answer is use the string type, it is appropriate
 to represent byte oriented data in python (also see the subject
 of this thread)


Everyone *is* saying to use the string type: the struct module doesn't
produce
structs.. it produces strings. The question is how to convert from Python
types
to byte-representations of said types in a string: you are saying (for
example) that
for shorts someone should use chr(1) etc. What if he needs to then send a
two
byte int from a Python int?

Just because he said his message was about a fairly simple requirement at
the moment, it doesn't mean that's the end of his problem. Everyone else
seems to have assumed that it'll advance as he supports more of this
protocol: that typelengthvariable data is apt to take on more specific
struct format messages based on type. So you might have a connection
request format that's pre-defined -- your example above -- or another which
needs to pass say, a 2 byte int instead.

The struct module easily accomplishes this: all together in one place.
Instead
of having to figure out how to calculate the 2 byte storage of an int down
the
road, or a 4 byte integer, etc. Not to mention endianness.

If the OP never ever has to do anything but the one specific problem he's
having
right now (and this isn't just an example he's making)  with really really
simple
data: two byte-ints and then some characters, then sure using chr() and
string
concatenation is probably fine. But struct works -perfectly- fine for
variable data
and as he extends his support for the protocol, is quite likely to be very
useful
to him: that's why we're suggesting it. IMHO.

I'd rather spend some trival time using string concatination (or the %
operator)
generating format strings for struct then have to write code for
int_to_2byte_int
and int_to_4byte_int and change_to_bigendian for when chr() doesn't
satisfy
the conversation needs enough.

Yeah, strings are exactly how he can and should store the byte oriented data
in Python 2.x. Using concatination and slicing + conversion to read and
parse a
byte stream coming off a line protocol... if you want to. But there's tools
that are
there that make it all alot easier too. The struct module for example.

--Stephen
--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[John Machin]

On Jan 26, 10:53 am, Martin v. Löwis mar...@v.loewis.de wrote:
  It deals with variable sized fields just fine:

  dtype = 18
  dlength = 32
  format = !BB%ds % dlength

  rawdata = struct.pack(format, (dtype,dlength,data))

 I wouldn't call this just fine, though - it involves
 a % operator to even compute the format string. IMO,
 it is *much* better not to use the struct module for this
 kind of problem, and instead rely on regular string
 concatenation.


IMO, it would be a good idea if struct.[un]pack supported a variable *
length operator that could appear anywhere that an integer constant
could appear, as in C's printf etc and Python's % formatting:

dlen = len(data)
rawdata = struct.pack(!BB*s, dtype, dlen, dlen, data)
# and on the other end of the wire:
dtype, dlen = struct.unpack(!BB, rawdata[:2])
data = struct.unpack(!*s, rawdata[2:], dlen)
# more than 1 count arg could be used if necessary
# *s would return a string
# *B, *H, *I, etc would return a tuple of ints in (3.X-speak)

I've worked with variable-length data that looked like
   len1, len2, len3, data1, data2, data3
and the * gadget would have been very handy:
len1, len2, len3 = unpack('!BBB', raw[:3])
data1, data2, data3 = unpack('!*H*i*d', raw[3:], len1, len2, len3)

Note the semantics of '!*H*i*d' would be different from '!8H2i7d'
because otherwise you'd need to do:
bundle = unpack('!*H*i*d', raw[3:], len1, len2, len3)
data1 = bundle[:len1]
data2 = bundle[len1:len1+len2]
data3 = bundle[len1+len2:]
--
http://mail.python.org/mailman/listinfo/python-list

Byte oriented data types in python

[Ravi]

I have following packet format which I have to send over Bluetooth.

packet_type (1 byte unsigned) || packet_length (1 byte unsigned) ||
packet_data(variable)

How to construct these using python data types, as int and float have
no limits and their sizes are not well defined.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Stephen Hansen]

I have following packet format which I have to send over Bluetooth.

 packet_type (1 byte unsigned) || packet_length (1 byte unsigned) ||
 packet_data(variable)

 How to construct these using python data types, as int and float have
 no limits and their sizes are not well defined.


Check out the struct module.

You want something like:

  data = struct.pack(BB4s, 1, 4, this)

It returns a string of bytes according to the specification -- B is unsigned
byte, and 4s is a string of 4 characters. The 1 feeds into the first byte,
the 4 into the second, etc.

--Stephen
--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[Martin v. Löwis]

 packet_type (1 byte unsigned) || packet_length (1 byte unsigned) ||
 packet_data(variable)
 
 How to construct these using python data types, as int and float have
 no limits and their sizes are not well defined.

In Python 2.x, use the regular string type: chr(n) will create a single
byte, and the + operator will do the concatenation.

In Python 3.x, use the bytes type (bytes() instead of chr()).

Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: Byte oriented data types in python

[skip]

Ravi packet_type (1 byte unsigned) || packet_length (1 byte unsigned) ||
Ravi packet_data(variable)

Ravi How to construct these using python data types, as int and float have
Ravi no limits and their sizes are not well defined.

Take a look at the struct and ctypes modules.

-- 
Skip Montanaro - s...@pobox.com - http://smontanaro.dyndns.org/
--
http://mail.python.org/mailman/listinfo/python-list

SQLAlchmey - new data types for PostgreSQL

[Kless]

It would be very interesting if the python community could to access
since SQLAlchemy to more features of the PostgreSQL 8.3 RDBMS
powerful, as are any new data types as: enumerated (ENUM) [1],  XML
[2], Universally Unique Identifiers (UUID) [3], and monetary [4].

For if anybody is interested on add them, see here: [5]


[1] http://www.postgresql.org/docs/8.3/static/datatype-enum.html
[2] http://www.postgresql.org/docs/8.3/static/datatype-xml.html
[3] http://www.postgresql.org/docs/8.3/static/datatype-uuid.html
[4] http://www.postgresql.org/docs/8.3/static/datatype-money.html

[5]
http://www.sqlalchemy.org/docs/05/sqlalchemy_databases_postgres.html

http://www.sqlalchemy.org/trac/browser/sqlalchemy/trunk/lib/sqlalchemy/databases/postgres.py
--
http://mail.python.org/mailman/listinfo/python-list

Re: Extending Python by Adding Keywords Data types

[Marc 'BlackJack' Rintsch]

On Wed, 01 Aug 2007 01:42:38 +, Maximus Decimus wrote:

 Since, I am an amateur in using python, could you please be more
 specific. For new data types, you had asked to implement the classes.
 I intend to use C for implementing these data types. So where do i
 need to implement these classes ie under which file or module in the
 python package.

Like Ben Finney said, first do something about the amateur status and work
through the tutorial.

Everything in Python is an object.  And classes are a way to define
templates for new objects.  After you know classes better, ask yourself
how you would implement the already existing data types with classes
yourself and then you'll get an idea how to create new data types.

If you want to implement new types in C there's a tutorial in the docs
called “Extending and Embedding” and a reference of the Python/C API.

But maybe you should start with implementing them in Python and only
convert those parts to C code that are *really* to slow in the prototype. 
Really means measure and don't guess.

A nice step between Python and C is Pyrex, a Python-like language with some
restrictions but also C data types.  Pyrex is then translated to C and can
be compiled as Python extension module.

Ciao,
Marc 'BlackJack' Rintsch
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Extending Python by Adding Keywords Data types

[Bjoern Schliessmann]

Maximus Decimus wrote:

 Since, I am an amateur in using python, 

http://en.wikipedia.org/wiki/Amateur
| Most commonly an amateur is understood to be someone who does
| something without pay or formal training. Conversely, a
| professional is someone who has received training in a particular
| area and who also makes a living from it.

So, you're not being paid coding in Python, and/or have
no official training. Many readers of this list share this
situation, me too.

 could you please be more specific. 

I return the question. Your statements are most vague. Please give
an example of what you try to realise.

 For new data types, you had asked to implement the classes. I
 intend to use C for implementing these data types. So where do i
 need to implement these classes ie under which file or module in
 the python package. 

Please do yourself a favor and read. Feel free to ask again for
specific problems.

http://docs.python.org/tut/node3.html
http://docs.python.org/tut/node11.html

Regards,


Björn
-- 
BOFH excuse #135:

You put the disk in upside down.

-- 
http://mail.python.org/mailman/listinfo/python-list