Happy Monday! I recently posted a tool which lets you import the request logs downloaded from the App Engine Admin Console into a sqlite3 database. This is useful for ad-hoc analysis and aggregation. It also understands the logs which you as a developer wrote with logging.error or similar methods--I am not sure if any of the existing Apache Combined Log Format parsers understand that extension, it would be great if they did. You can also describe a custom column on those logs.
You can view it at http://code.google.com/p/google-app-engine-samples/source/browse/trunk/logparser/logparser.py You can download using a subversion client, or directly at http://google-app-engine-samples.googlecode.com/svn/trunk/logparser/logparser.py Here are some examples from the docstring of the tool: Example: # Download request logs into requests.txt appcfg.py request_logs --severity 0 --include_all <appdirectory> requests.txt # Run the logparser to insert them into requests.db. # (You can specify multiple input files.) logparser.py --db requests.db requests.txt # Query them using the sqlite3 interactive command line interface. sqlite3 requests.db sqlite> -- what are the most common 404s? sqlite> select distinct request_line, count(*) from requests ...> where status == 404 group by request_line order by request_line desc; sqlite> -- what requests see 'Deadline Exceeded'? sqlite> select distinct request_line from requests ...> where applog like '%DeadlineExceeded%'; sqlite> -- How many loading requests were seen? sqlite> select count(*) from requests where loading_request=1; sqlite> -- What was the average cpm across all pages? sqlite> select sum(cpm_usd)/count(cpm_usd) from requests; The flag --custom_column "widgets:^1:[0-9.]+ Found ([0-9]+) widgets" will locate all Info logs (severity 1) which look like this: 1:1286325423.286856 Found 12 widgets and pull out the "12" part. Now you can run a query like these: sqlite> -- How many widgets were seen? sqlite> select select sum(widgets) from requests; sqlite> -- What was the approximate the processing time per widget? sqlite> select sum(cpu_ms)/cast(sum(widgets) as float) ...> from requests where widgets > ''; Let me know if you find this tool useful. --Matthew -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
