Re: [Tutor] Counting and grouping dictionary values in Python 2.7

[Michael Selik]

On Fri, Jul 8, 2016 at 10:15 AM Bruce Dykes  wrote:

> I'm compiling application logs from a bunch of servers, reading the log
> entries, parsing each log entry into a dictionary, and compiling all the
> log entries into a single list of dictionaries.
>

Seems reasonable. Perhaps instead of having each log entry as a dictionary,
you might prefer to create a namedtuple.


> Now, what I need to do with this arbitrarily count and total the values in
> the dictionaries, ie the total amount and number of items for transaction
> id 387, or the total number of crackers sold in NJ stores.


Use the Counter class from the collections module. Loop over your records,
write an if-statement to determine which records are interesting, then
increment the count for the appropriate key.

If you're having trouble, share your code and the error message.

Also, is there any particular advantage to pickling the list and having two
> files, one, the pickled file to be read as a data source, and the .csv file
> for portability/readability, as opposed to just a single .csv file that
> gets reparsed by the reporting script?
>

Probably simpler to keep just a CSV.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Counting and grouping dictionary values in Python 2.7

[Bruce Dykes]

On Fri, Jul 8, 2016 at 1:33 PM, Alan Gauld via Tutor 
wrote:

> On 08/07/16 14:22, Bruce Dykes wrote:
>
> > with it is writing the list of dictionaries to a .csv file, and to date,
> > we've been able to get by doing some basic analysis by simply using grep
> > and wc, but I need to do more with it now.
>
> I'm a big fan of using the right tool for the job.
> If you got your data in CSV have you considered using a
> spreadsheet to read the data and analyse it? They have lots
> of formulae and stats functions built in and can do really
> cool graphs etc and can read csv files natively.
>
> Python might be a better tool if you want regular identical reports, say
> on a daily basis, but for ad-hoc analysis, or at least till you know
> exactly what you need, Excel or Calc are possibly better tools.
>
>
>
We can and have used spreadsheets for small ad-hoc things, but no, we need
two things, first, as noted, a daily report with various basic analyses,
mainly totals, and percentages, and second, possibly, some near-current
alarm checks, depending. That's less important, actually, but it might be a
nice convenience. In the first instance, we want the reports to be accessed
and displayed as web pages. Now, likewise, I'm sure there's a CMS that
might make semi-quick work of this as well, but really, all I need to do is
to display some web pages and run some cgi scripts.

bkd
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Counting and grouping dictionary values in Python 2.7

[Alan Gauld via Tutor]

On 08/07/16 14:22, Bruce Dykes wrote:

> with it is writing the list of dictionaries to a .csv file, and to date,
> we've been able to get by doing some basic analysis by simply using grep
> and wc, but I need to do more with it now.

I'm a big fan of using the right tool for the job.
If you got your data in CSV have you considered using a
spreadsheet to read the data and analyse it? They have lots
of formulae and stats functions built in and can do really
cool graphs etc and can read csv files natively.

Python might be a better tool if you want regular identical reports, say
on a daily basis, but for ad-hoc analysis, or at least till you know
exactly what you need, Excel or Calc are possibly better tools.

> Also, is there any particular advantage to pickling the list and having two
> files, one, the pickled file to be read as a data source, and the .csv file

Probably not, the cost of converting the strings in the csv to objects
is not that high unless you have huge volumes to manage(ie millions of
records).


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Counting and grouping dictionary values in Python 2.7

[Bruce Dykes]

I'm compiling application logs from a bunch of servers, reading the log
entries, parsing each log entry into a dictionary, and compiling all the
log entries into a single list of dictionaries. At present, all I'm doing
with it is writing the list of dictionaries to a .csv file, and to date,
we've been able to get by doing some basic analysis by simply using grep
and wc, but I need to do more with it now.

Here's what the data structures look like:

NY = ['BX01','BX02','BK01','MN01','SI01']
NJ = ['NW01','PT01','PT02']
CT = ['ST01','BP01','NH01']

sales = [
{'store':'store','date':'date','time':'time','state':'state',transid':'transid','product':'product','price':'price'},
{'store':'BX01','date':'8','time':'08:55','state':'NY',transid':'387','product':'soup','price':'2.59'},
{'store':'NW01','date':'8','time':'08:57','state':'NJ',transid':'24','product':'apples','price':'1.87'},
{'store':'BX01','date':'8','time':'08:56','state':'NY',transid':'387','product':'crackers','price':'3.44'}]

The first group of list with the state abbreviations is there to add the
state information to the compiled log, as it's not included in the
application log. The first dictionary in the list, with the duplicated key
names in the value field is there to provide a header line as the first
line in the compiled .csv file.

Now, what I need to do with this arbitrarily count and total the values in
the dictionaries, ie the total amount and number of items for transaction
id 387, or the total number of crackers sold in NJ stores. I think the
collections library has the functions I need, but I haven't been able to
grok the examples uses I've seen online. Likewise, I know I could build a
lot of what I need using regex and lists, etc, but if Python 2.7 already
has the blocks there to be used, well let's use the blocks then.

Also, is there any particular advantage to pickling the list and having two
files, one, the pickled file to be read as a data source, and the .csv file
for portability/readability, as opposed to just a single .csv file that
gets reparsed by the reporting script?

Thanks in advance
bkd
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor