[issue1818] Add named tuple reader to CSV module

[Raymond Hettinger]

Change by Raymond Hettinger :


--
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Skip Montanaro]

Skip Montanaro  added the comment:

FWIW, I relinquished my check-in privileges quite awhile ago. This should
almost certainly no longer be assigned to me.

S

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Raymond Hettinger]

Change by Raymond Hettinger :


--
priority: low -> normal
versions: +Python 3.8 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Cédric Krier]

Changes by Cédric Krier cedric.kr...@b2ck.com:


--
nosy: +ced

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Skip Montanaro]

Skip Montanaro added the comment:

I looked at this six years ago. I still haven't found a situation where I pined 
for a NamedTupleReader. That said, I have no objection to committing it if 
others, more well-versed in current Python code and NamedTuples than I gives it 
a pass. Note that I added a couple comments to the csv.py diff, but nobody 
either updated the code or explained why I was out in the weeds in my comments.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Ilia Kurenkov]

Ilia Kurenkov added the comment:

Friendly reminder that this exists.

I know everyone's busy and this is marked as low-priority, but I'm gonna keep 
bumping this till we add a solution :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Raymond Hettinger]

Raymond Hettinger added the comment:

Skip or Barry, do you want to look at this?

--
assignee:  - skip.montanaro
nosy: +skip.montanaro
stage: needs patch - patch review
versions:  -Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Ilia Kurenkov]

Ilia Kurenkov added the comment:

As my contribution during the sprints at PyCon 2015, I've tweaked Jervis's 
patch a little and updated the tests/docs to work with Python 3.5.

My only real change was placing the basic reader object inside a generator 
expression that filters out empty lines. Being partial to functional 
programming I find this removes some of the code clutter in __next__(), letting 
that method focus on turning rows into tuples.

Hopefully this will rekindle the discussion!

--
nosy: +copper-head
versions: +Python 3.5
Added file: http://bugs.python.org/file39139/1818_py35.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Daniel Lenski]

Daniel Lenski added the comment:

Here's the class I have been using for reading namedtuples from CSV files:

from collections import namedtuple
from itertools import imap
import csv

class CsvNamedTupleReader(object):
__slots__ = ('_r', 'row', 'fieldnames')
def __init__(self, *args, **kwargs):
self._r = csv.reader(*args, **kwargs)
self.row = namedtuple(row, self._r.next())
self.fieldnames = self.row._fields

def __iter__(self):
#FIXME: how about this? return imap(self.row._make, 
self._r[:len(self.fieldnames)]
return imap(self.row._make, self._r)

dialect = property(lambda self: self._r.dialect)
line_num = property(lambda self: self._r.line_num)

This class wraps csv.reader since it doesn't seem to be possible to inherit 
from it. It uses itertools.imap to iterate over the rows output by csv.reader 
and convert them to the namedtuple class.

One thing that needs fixing (marked with FIXME above) is what to do in the case 
of a row which has more fields than the header row. The simplest solution is 
simply to truncate such a row, but perhaps more options are needed, similar to 
those offered by DictReader.

--
nosy: +dlenski

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Mark Lawrence]

Changes by Mark Lawrence breamore...@yahoo.co.uk:


--
nosy:  -BreamoreBoy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Andrew Svetlov]

Changes by Andrew Svetlov andrew.svet...@gmail.com:


--
nosy: +asvetlov

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Ahsan Nawroj]

Changes by Ahsan Nawroj ahsan.naw...@gmail.com:


--
nosy: +ainur0160

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Ahsan Nawroj]

Changes by Ahsan Nawroj ahsan.naw...@gmail.com:


--
nosy:  -ainur0160

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

Unassigning, this needs fresh thought and a fresh patch from someone who can 
devote a little deep thinking on how to solve this problem cleanly.  In the 
meantime, it is no problem to simply cast the CSV tuples into named tuples.

--
assignee: rhettinger - 
priority: normal - low
stage: patch review - needs patch
versions:  -Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Mark Lawrence]

Mark Lawrence breamore...@yahoo.co.uk added the comment:

Closing as no response to msg110598.

--
status: pending - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

Re-opening because we ought to do something along these lines at some point.  
The DictReader and DictWriter are inadequate for preserving order and they are 
unnecessarily memory intensive (one dict per record).  

FWIW, the non-conforming field name problem has already been solved by recent 
improvements to collections.namedtuple using rename=True.

--
assignee: barry - rhettinger
status: closed - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Mark Lawrence]

Mark Lawrence breamore...@yahoo.co.uk added the comment:

I suggest that this is closed unless anyone shows an active interest in it.

--
nosy: +BreamoreBoy
status: open - pending
versions: +Python 3.2, Python 3.3 -Python 2.7, Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Skip Montanaro]

Changes by Skip Montanaro s...@pobox.com:


--
nosy:  -skip.montanaro

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

See also this python-ideas thread: 
http://mail.python.org/pipermail/python-ideas/2010-April/006991.html

--
nosy: +merwok

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Skip Montanaro]

Skip Montanaro s...@pobox.com added the comment:

Type conversion is a whole 'nuther kettle of fish.  This particular thread is 
long and complex enough that it shouldn't be made more complex.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

I don't understand why NamedTupleReader requires the fieldnames array
rather than the namedtuple class itself. If you could pass it the
namedtuple class, users could choose whatever namedtuple subclass with
whatever additional methods or behaviour suits them. It would make
NamedTupleReader more flexible and more useful.

--
message_count: 23.0 - 24.0
nosy: +pitrou
nosy_count: 5.0 - 6.0

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Skip Montanaro]

Skip Montanaro s...@pobox.com added the comment:

I find this aspect of the proposal disturbing:

 If *fieldnames* is None the values in the 
first row of the *csvfile* will be used as the fieldnames

I don't think this should be implicit.  It makes the NamedTupleReader
semantically different from the vanilla reader object which treats
every row of the file as data.  The use of the fieldnames argument
should be required for NamedTupleReader objects as it is for DictReader 
objects.

--
message_count: 25.0 - 26.0

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Skip Montanaro]

Changes by Skip Montanaro s...@pobox.com:


--
message_count: 26.0 - 25.0

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Skip Montanaro]

Skip Montanaro s...@pobox.com added the comment:

I retract my previous comment.  I don't use the DictReader the way it
operates (fieldnames==None = first row is a header) and forgot about
that behavior.

--
message_count: 25.0 - 26.0

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Jervis Whitley]

Jervis Whitley jervi...@gmail.com added the comment:

Jervis in csv.rst removed reference to reader.next() as a public method.

Skip Because?  I've not seen any discussion in this issue or in any
Skip other forums
Skip (most certainly not on the c...@python.org mailing list) which
would Skip suggest
Skip that csv.reader's next() method should no longer be a public method.

I agree, this should be applied separately.

--
message_count: 26.0 - 27.0
Added file: http://bugs.python.org/file13268/ntreader5_py3_1.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Jervis Whitley]

Jervis Whitley jervi...@gmail.com added the comment:

Antoine I don't understand why NamedTupleReader requires the 
Antoine fieldnames array
Antoine rather than the namedtuple class itself. If you could pass it
Antoine the namedtuple class, users could choose whatever namedtuple 
Antoine subclass with whatever additional methods or behaviour suits
Antoine them. It would make NamedTupleReader more flexible and more 
Antoine useful.

The NamedTupleReader does take the namedtuple class as the fieldnames
argument. It can be a namedtuple, a 'fieldnames' array or None. 
If a namedtuple is used as the fieldnames argument, returned rows are
created using ._make from the this namedtuple. Unless I have read your
requirements incorrectly, this is the behaviour you describe.

Given the confusion, I accept that the documentation needs to be improved. 

The NamedTupleReader and Writer were created to follow as closely as
possible the behaviour (and signature) of the DictReader and DictWriter,
with the exception of using namedtuples instead of dicts.

--
message_count: 27.0 - 28.0

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

Ok, I got misled by the documentation (The contents of *fieldnames* are
passed directly to be used as the namedtuple fieldnames), and your
implementation is a bit difficult to follow.

--
message_count: 28.0 - 29.0

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Jervis Whitley]

Changes by Jervis Whitley jervi...@gmail.com:


Removed file: http://bugs.python.org/file13268/ntreader5_py3_1.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Jervis Whitley]

Changes by Jervis Whitley jervi...@gmail.com:


Added file: http://bugs.python.org/file13275/ntreader6_py27.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Jervis Whitley]

Jervis Whitley jervi...@gmail.com added the comment:

Updated version of docs for 2.7 and 3k.

--
message_count: 29.0 - 30.0
Added file: http://bugs.python.org/file13274/ntreader6_py3.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Jervis Whitley]

Jervis Whitley jervi...@gmail.com added the comment:

Added a patch against py3k branch.

in csv.rst removed reference to reader.next() as a public method.

Added file: http://bugs.python.org/file13263/ntreader4_py3_1.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Skip Montanaro]

Skip Montanaro s...@pobox.com added the comment:

Jervis in csv.rst removed reference to reader.next() as a public method.

Because?  I've not seen any discussion in this issue or in any other forums
(most certainly not on the c...@python.org mailing list) which would suggest
that csv.reader's next() method should no longer be a public method.

Skip

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

The two latest patches (ntreader4.diff and
named_tuple_write_header.patch) seem like they are going in the right
direction and are getting close.

Barry or Skip, is this something you want in your module?

--
stage:  - patch review
type:  - feature request
versions: +Python 2.7, Python 3.1 -Python 2.6

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Skip Montanaro]

Skip Montanaro s...@pobox.com added the comment:

Raymond Barry or Skip, is this something you want in your module?

Sorry, I haven't really looked at this ticket other than to notice its
presence.  I wrote the DictReader/DictWriter functions way back when, so I'm
pretty comfortable using them.  I haven't felt the need for any other reader
or writer which manipulates file headers.

Skip

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Barry A. Warsaw]

Barry A. Warsaw ba...@python.org added the comment:

I think it would be useful to have.

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Skip Montanaro]

Skip Montanaro s...@pobox.com added the comment:

Hrm... I replied twice by email.  Only one comment appears to have
survived the long trip.  Here's my second reply:


Rob NamedTupleReader and NamedTupleWriter should be inverses.  This
Rob means that NamedTupleWriter needs to write headers.  This should
Rob produce identical output as the dict writer example, but it's much
Rob cleaner.

You're assuming that one instance of these classes will read or write an
entire file.  What if you want to append lines to an existing CSV file or
pick up reading a file with a new reader which has already be partially
processed?

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Skip Montanaro]

Skip Montanaro s...@pobox.com added the comment:

Let me be more explicit.  I don't know how it implements it, but I think
you really need to give the user the option of specifying the field
names and not reading/writing headers.  It can't be implicit as I
interpreted Rob's earlier comment:

 NamedTupleReader and NamedTupleWriter should be inverses.
 This means that NamedTupleWriter needs to write headers.

Skip

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Jervis Whitley]

Jervis Whitley jervi...@gmail.com added the comment:

Skip Let me be more explicit.  I don't know how it implements it, but I 
think
Skip you really need to give the user the option of specifying the 
field
Skip names and not reading/writing headers.  It can't be implicit as I
Skip interpreted Rob's earlier comment:

rrenaud NamedTupleReader and NamedTupleWriter should be inverses.
rrenaud This means that NamedTupleWriter needs to write headers.

I agree with Skip, we mustn't have a 'wroteheader' flag internal to the 
NamedTupleWriter.

Currently to write a 'header' row with a csv.writer you could (for 
example) pass a tuple of header names to writerow. NamedTupleWriter
is no different, you would have a namedtuple of header names instead of
a tuple of header names.

I would not like to see another flag added to the initialisation process
to enable the writing of a header row as the 'first' (or any) row 
written to a file.  We could add a function 'writeheader' that would
write the contents of 'fieldnames' as a row, but I don't like the idea.

Cheers,

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Rob Renaud]

Rob Renaud rren...@google.com added the comment:

I want to make sure I understand.  Am I correct in believing that Skip
thinks writing headers should be optional, while Jervis believes we
should leave the burden to the NamedTupleWriter client?  

I agree that we should not unconditionally write headers, but I think
that we should write headers by default, much like we read them by default.

I believe the implicit header writing is very elegant, and the only
reason that the DictWriter object doesn't write headers is the impedance
mismatch between dicts and CSV.  namedtuples has the field order
information, the impedance mismatch is gone, we should no longer be
hindered.  Implicitly reading but not explicitly writing headers just
seems wrong.

It also seems wrong to require the construction of header namedtuple
objects.  It's much less natural than dicts holding identity mappings.

 Point._make(Point._fields)
Point(x='x', y='y')

To me, that just looks weird and non-obvious to me.  That Point instance
doesn't really fit in my mind as something that should be a Point.

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Skip Montanaro]

Skip Montanaro s...@pobox.com added the comment:

More concretely, I don't think this is so onerous:

names = [col1, col2, color]
writer = csv.DictWriter(open(f.csv, wb), fieldnames=names, ...)
writer.writerow(dict(zip(names, names)))
...

or

f = open(f.csv, rb)
names = csv.reader(f).next()
reader = csv.DictReader(f, fieldnames=names, ...)
...

Skip

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Rob Renaud]

Rob Renaud rren...@google.com added the comment:

I did a search on Google code for the DictReader constructor.  I
analyzed the first 3 pages, the fieldnames parameter was used in 14 of
27 cases (discounting unittest code built into Python) and was not
used in 13 of 27 cases.  I suppose that means headered csv files are
sufficiently rare that they shouldn't be created implicitly by
default.  I still don't like the lack of symmetry of supporting
implicit header reads, but not implicit header writes.

On Thu, Feb 26, 2009 at 8:00 PM, Skip Montanaro rep...@bugs.python.org wrote:

 Skip Montanaro s...@pobox.com added the comment:

 More concretely, I don't think this is so onerous:

names = [col1, col2, color]
writer = csv.DictWriter(open(f.csv, wb), fieldnames=names, ...)
writer.writerow(dict(zip(names, names)))
...

 or

f = open(f.csv, rb)
names = csv.reader(f).next()
reader = csv.DictReader(f, fieldnames=names, ...)
...

 Skip

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue1818
 ___


___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

 I don't think you should write them by default.  
 I've worked with lots of CSV files which have no headers. 

My experience has been the same as Skips.

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Skip Montanaro]

Skip Montanaro s...@pobox.com added the comment:

Rob I still don't like the lack of symmetry of supporting implicit
Rob header reads, but not implicit header writes.

A header is nothing more than a row in the CSV file with special
interpretation applied by the user.  There is nothing implicit about it.
If you know the first line is a header, use the recipe I posted.  If not,
supply your own fieldnames and treat the first row as data.

Skip

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Rob Renaud]

Rob Renaud rren...@google.com added the comment:

I am totally new to Python dev.  I reinvented a NamedTupleReader
tonight, only to find out that it was created a year ago.  My primary
motivation is that DictReader reads headers nicely, but DictWriter
totally sucks at handling them.

Consider doing some filtering on a csv file, like so.

sample_data = [
'title,latitude,longitude',
'OHO Ofner  Hammecke Reinigungsgesellschaft mbH,48.128265,11.610848',
'Kitchen Kaboodle,45.544241,-122.715728',
'Walgreens,28.339727,-81.596367',
'Gurnigel Pass,46.731944,7.447778'
]

def filter_with_dict_reader_writer():
  accepted_rows = []
  for row in csv.DictReader(sample_data):
if float(row['latitude'])  0.0 and float(row['longitude'])  0.0:
  accepted_rows.append(row)

  field_names = csv.reader(sample_data).next()
  output_writer = csv.DictWriter(open('accepted_by_dict.csv', 'w'),
 field_names)
  output_writer.writerow(dict(zip(field_names, field_names)))
  output_writer.writerows(accepted_rows)

You have to work so hard to maintain the headers when you write the file
with DictWriter.  I understand this is a limitation of dicts throwing
away the order information.  But namedtuples don't have that problem.

NamedTupleReader and NamedTupleWriter should be inverses.  This means
that NamedTupleWriter needs to write headers.  This should produce
identical output as the dict writer example, but it's much cleaner.

def filter_with_named_tuple_reader_writer():
   accepted_rows = []
   for row in csv.NamedTupleReader(sample_data):
 if float(row.latitude)  0.0 and float(row.longitude)  0.0:
   accepted_rows.append(row)

   output_writer = csv.NamedTupleWriter(
   open('accepted_by_named_tuple.csv', 'w'))
   output_writer.writerows(accepted_rows)

I patched on top of the existing NamedTupleWriter patch adding support
for writing headers.  I don't know if that's bad style/etiquette, etc.

--
nosy: +rrenaud
Added file: http://bugs.python.org/file13187/named_tuple_write_header.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Rob Renaud]

Rob Renaud rren...@google.com added the comment:

My previous patch could write the header twice.  But I am not sure about
about how the writer should handle the fieldnames parameter on one hand,
and the namedtuple._fields on the other.

Added file: http://bugs.python.org/file13188/named_tuple_write_header2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Rob Renaud]

Changes by Rob Renaud rren...@google.com:


Removed file: http://bugs.python.org/file13187/named_tuple_write_header.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Jervis Whitley]

Jervis Whitley jervi...@gmail.com added the comment:

Updated NamedTupleReader to give a rename=False keyword argument.
rename is passed directly to the namedtuple factory function to enable
automatic handling of invalid fieldnames.

Two new tests for the rename keyword.

Cheers,

Added file: http://bugs.python.org/file13009/ntreader4.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Jervis Whitley]

Jervis Whitley jervi...@gmail.com added the comment:

An implementation of a namedtuple reader and writer.

Created a writer for the case where user would like to specify
desired field names and default values on missing field names.

e.g.
mywriter = NamedTupleWriter(f, fieldnames=['f1', 'f2', 'f3'], 
restval='missing')

Nt = namedtuple('LessFields', 'f1 f3')
nt = Nt(f1='one', f2=2)

mywriter.writerow(nt) # writes one,missing,2

any thoughts on case where defined fieldname has a leading 
underscore? Should there be a flag to silently ignore? 

e.g. 
if self._ignore_underscores:
   fieldname = fieldname.lstrip('_')

Leading underscores may be present in an unsighted csv file,
additionally, spaces and other non alpha numeric characters pose 
a problem that does not affect the DictReader class. 

Cheers,

--
keywords: +patch
nosy: +jdwhitley
Added file: http://bugs.python.org/file12990/ntreader3.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

Consider providing a hook to a function that converts non-conforming
field names (ones with a leading underscore, leading digit, non-letter,
keyword, or duplicate name).

class NamedTupleReader:
def __init__(self, f, fieldnames=None, restkey=None, restval=None,
 dialect=excel, fieldnamer=None, *args, **kwds):
 . . .

I'm going to either post a recipe to do the renaming or provide a static
method for the same purpose.   It might work like this:

   renamer(['abc', 'def', '1', '_hidden', 'abc', 'p', 'abc'])
  ['abc', 'x_def', 'x_1', 'x_hidden', 'x_abc', 'p', 'x1_abc']

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

In r69480, named tuples gained the ability to automatically rename
invalid fieldnames.

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1818
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Raymond Hettinger]

Raymond Hettinger added the comment:

Barry, any thoughts on this?

__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1818
__
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Skip Montanaro]

Skip Montanaro added the comment:

I'd personally be kind of surprised if Barry had any thoughts on this.
Is there any reason this couldn't be pushed down into the C code and
replace the normal tuple output completely?  In the absence of any
fieldnames you could just dream some up, like field001, field002,
etc.

Skip

--
nosy: +skip.montanaro

__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1818
__
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1818] Add named tuple reader to CSV module

[Raymond Hettinger]

New submission from Raymond Hettinger:

Here's a proof-of-concept patch.  If approved, will change from
generator form to match the other readers and will add a test suite.

The idea corresponds to what is currently done by the dict reader but
returns a space and time efficient named tuple instead of a dict.  Field
order is preserved and named attribute access is supported.

A writer is not needed because named tuples can be feed into the
existing writer just like regular tuples.

--
assignee: barry
components: Library (Lib)
files: ntreader.diff
messages: 59866
nosy: barry, rhettinger
severity: normal
status: open
title: Add named tuple reader to CSV module
versions: Python 2.6
Added file: http://bugs.python.org/file9151/ntreader.diff

__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1818
__
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com