Re: [ANNOUNCE] libtraceevent.git
On Mon, Oct 12, 2020 at 11:19:50AM -0400, Steven Rostedt wrote: > Once it's shown that it works for all the package maintainers, I will tag > it which should create the tarballs automatically on the above link. Hi. It builds fine for me after manually creating the tarball from git. Once there is an official versioned tarball I'll push it into openSUSE. I presume some perf Makefile changes will be forthcoming to use it, rather than continuing to force build it out of TRACE_EVENT_DIR Tony
Re: [PATCH 1/2] perf script python: integrate page reclaim analyze script
On 9/25/19 6:56 PM, Tony Jones wrote: > On 9/18/19 7:38 AM, Yafang Shao wrote: >> A new perf script page-reclaim is introduced in this patch. This new script >> is used to report the page reclaim details. The possible usage of this >> script is as bellow, >> - identify latency spike caused by direct reclaim >> - whehter the latency spike is relevant with pageout >> - why is page reclaim requested, i.e. whether it is because of memory >> fragmentation >> - page reclaim efficiency >> etc >> In the future we may also enhance it to analyze the memcg reclaim. >> >> Bellow is how to use this script, >> # Record, one of the following >> $ perf record -e 'vmscan:mm_vmscan_*' ./workload >> $ perf script record page-reclaim >> >> # Report >> $ perf script report page-reclaim >> >> # Report per process latency >> $ perf script report page-reclaim -- -p > > > I tested it with global-dhp__pagereclaim-performance from mmtests and got > what appears to be reasonable results and the output looks correct and > useful. However I'm not a vm expert so I can't comment further. Hopefully > someone on linux-mm can give more specific feedback. > > There is one issue with Python3, see below. I didn't test with Python2. Ok, I guess this wasn't actually tested with Python3 as itervalues() is Python2 only. Any scripts need to work with both Python2.6+ and Python3. # perf script -i /tmp/perf.out -s page-reclaim.py -- -p ... Traceback (most recent call last): File "page-reclaim.py", line 305, in trace_end i.display_proc(), File "page-reclaim.py", line 268, in display_proc print_proc_latency(sorted(self.stat.stats['latency'].itervalues(), AttributeError: 'dict' object has no attribute 'itervalues' Fatal Python error: problem in Python trace event handler Use a try/except to handle this. Thanks Tony
Re: [PATCH 1/2] perf script python: integrate page reclaim analyze script
On 9/18/19 7:38 AM, Yafang Shao wrote:
> A new perf script page-reclaim is introduced in this patch. This new script
> is used to report the page reclaim details. The possible usage of this
> script is as bellow,
> - identify latency spike caused by direct reclaim
> - whehter the latency spike is relevant with pageout
> - why is page reclaim requested, i.e. whether it is because of memory
> fragmentation
> - page reclaim efficiency
> etc
> In the future we may also enhance it to analyze the memcg reclaim.
>
> Bellow is how to use this script,
> # Record, one of the following
> $ perf record -e 'vmscan:mm_vmscan_*' ./workload
> $ perf script record page-reclaim
>
> # Report
> $ perf script report page-reclaim
>
> # Report per process latency
> $ perf script report page-reclaim -- -p
I tested it with global-dhp__pagereclaim-performance from mmtests and got what
appears to be reasonable results and the output looks correct and useful.
However I'm not a vm expert so I can't comment further. Hopefully someone on
linux-mm can give more specific feedback.
There is one issue with Python3, see below. I didn't test with Python2.
>
> + @classmethod
> + def shrink_inactive(cls, pid, scanned, reclaimed, flags):
> + event = cls.events.get(pid)
> + if event and event.tracing():
> + # RECLAIM_WB_ANON 0x1
> + # RECLAIM_WB_FILE 0x2
> + _type = (flags & 0x2) >> 1
> + event.process_lru(lru[_type], scanned, reclaimed)
> +
> + @classmethod
> + def writepage(cls, pid, flags):
> + event = cls.events.get(pid)
> + if event and event.tracing():
> + # RECLAIM_WB_ANON 0x1
> + # RECLAIM_WB_FILE 0x2
> + # RECLAIM_WB_SYNC 0x4
> + # RECLAIM_WB_ASYNC 0x8
> + _type = (flags & 0x2) >> 1
> + _io = (flags & 0x4) >> 2
> +
> + event.process_writepage(lru[_type], sync_io[_io])
> +
> +@classmethod
Space indentation on line above. For python3 this results in:
File "tools/perf/scripts/python/page-reclaim.py", line 217
@classmethod
^
TabError: inconsistent use of tabs and spaces in indentation
> + def iterate_proc(cls):
> + if show_opt != Show.DEFAULT:
> + print("\nPer process latency (ms):")
> + print_proc_latency(latency_metric, 'pid', '[comm]')
> +
> + if show_opt == Show.VERBOSE:
> + print("%20s %s" % ('timestamp','latency(ns)'))
Thanks
Tony
Re: [PATCH 2/2] tracing, vmscan: add comments for perf script page-reclaim
On 9/18/19 7:38 AM, Yafang Shao wrote: > Currently there's no easy way to make perf scripts in sync with > tracepoints. One possible way is to run perf's tests regularly, another way > is once we changes the definitions of tracepoints we must keep in mind that > the perf scripts which are using these tracepoints must be changed as well. > So I add this comment for the new introduced page-reclaim script as a > reminder. > > Signed-off-by: Yafang Shao This seems like the tail wagging the dog to me. If we added this to every tp defn that is used by a perf perl/python script the tp headers would be littered with such comments.
[tip:perf/urgent] perf script python: Add Python3 support to export-to-sqlite.py
Commit-ID: ebf6c5c181abe9309788c6241d39602a1ce18723
Gitweb: https://git.kernel.org/tip/ebf6c5c181abe9309788c6241d39602a1ce18723
Author: Tony Jones
AuthorDate: Fri, 8 Mar 2019 16:05:17 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Mon, 11 Mar 2019 16:12:59 -0300
perf script python: Add Python3 support to export-to-sqlite.py
Support both Python2 and Python3 in the export-to-sqlite.py script
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Acked-by: Adrian Hunter
Link: http://lkml.kernel.org/r/20190309000518.2438-4-to...@suse.de
Signed-off-by: Seeteena Thoufeek
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/export-to-sqlite.py | 23 ++-
1 file changed, 14 insertions(+), 9 deletions(-)
diff --git a/tools/perf/scripts/python/export-to-sqlite.py
b/tools/perf/scripts/python/export-to-sqlite.py
index eb63e6c7107f..3da338243aed 100644
--- a/tools/perf/scripts/python/export-to-sqlite.py
+++ b/tools/perf/scripts/python/export-to-sqlite.py
@@ -10,6 +10,8 @@
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
+from __future__ import print_function
+
import os
import sys
import struct
@@ -60,11 +62,14 @@ perf_db_export_mode = True
perf_db_export_calls = False
perf_db_export_callchains = False
+def printerr(*args, **keyword_args):
+ print(*args, file=sys.stderr, **keyword_args)
+
def usage():
- print >> sys.stderr, "Usage is: export-to-sqlite.py
[] [] []"
- print >> sys.stderr, "where:columns 'all' or 'branches'"
- print >> sys.stderr, " calls 'calls' => create calls
and call_paths table"
- print >> sys.stderr, " callchains 'callchains' => create
call_paths table"
+ printerr("Usage is: export-to-sqlite.py []
[] []");
+ printerr("where:columns 'all' or 'branches'");
+ printerr(" calls 'calls' => create calls and
call_paths table");
+ printerr(" callchains 'callchains' => create
call_paths table");
raise Exception("Too few arguments")
if (len(sys.argv) < 2):
@@ -100,7 +105,7 @@ def do_query_(q):
return
raise Exception("Query failed: " + q.lastError().text())
-print datetime.datetime.today(), "Creating database..."
+print(datetime.datetime.today(), "Creating database ...")
db_exists = False
try:
@@ -378,7 +383,7 @@ if perf_db_export_calls:
call_query.prepare("INSERT INTO calls VALUES (?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?)")
def trace_begin():
- print datetime.datetime.today(), "Writing records..."
+ print(datetime.datetime.today(), "Writing records...")
do_query(query, 'BEGIN TRANSACTION')
# id == 0 means unknown. It is easier to create records for them than
replace the zeroes with NULLs
evsel_table(0, "unknown")
@@ -397,14 +402,14 @@ unhandled_count = 0
def trace_end():
do_query(query, 'END TRANSACTION')
- print datetime.datetime.today(), "Adding indexes"
+ print(datetime.datetime.today(), "Adding indexes")
if perf_db_export_calls:
do_query(query, 'CREATE INDEX pcpid_idx ON calls
(parent_call_path_id)')
do_query(query, 'CREATE INDEX pid_idx ON calls (parent_id)')
if (unhandled_count):
- print datetime.datetime.today(), "Warning: ", unhandled_count,
" unhandled events"
- print datetime.datetime.today(), "Done"
+ print(datetime.datetime.today(), "Warning: ", unhandled_count,
" unhandled events")
+ print(datetime.datetime.today(), "Done")
def trace_unhandled(event_name, context, event_fields_dict):
global unhandled_count
[tip:perf/urgent] perf script python: Add Python3 support to exported-sql-viewer.py
Commit-ID: beda0e725e5f06aca27eda2434ea9447dad88e36
Gitweb: https://git.kernel.org/tip/beda0e725e5f06aca27eda2434ea9447dad88e36
Author: Tony Jones
AuthorDate: Fri, 8 Mar 2019 16:05:15 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Mon, 11 Mar 2019 16:12:52 -0300
perf script python: Add Python3 support to exported-sql-viewer.py
Support both Python2 and Python3 in the exported-sql-viewer.py script.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Acked-by: Adrian Hunter
Link: http://lkml.kernel.org/r/20190309000518.2438-2-to...@suse.de
Signed-off-by: Seeteena Thoufeek
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/exported-sql-viewer.py | 42
1 file changed, 28 insertions(+), 14 deletions(-)
diff --git a/tools/perf/scripts/python/exported-sql-viewer.py
b/tools/perf/scripts/python/exported-sql-viewer.py
index afec9479ca7f..e38518cdcbc3 100755
--- a/tools/perf/scripts/python/exported-sql-viewer.py
+++ b/tools/perf/scripts/python/exported-sql-viewer.py
@@ -88,11 +88,20 @@
#
7fab593ea956 48 89 15 3b 13 22 00movq %rdx,
0x22133b(%rip)
# 8107675243232 2ls 22011 22011 hardware interrupt No
7fab593ea956 _dl_start+0x26 (ld-2.19.so) -> 86a012e0 page_fault
([kernel])
+from __future__ import print_function
+
import sys
import weakref
import threading
import string
-import cPickle
+try:
+ # Python2
+ import cPickle as pickle
+ # size of pickled integer big enough for record size
+ glb_nsz = 8
+except ImportError:
+ import pickle
+ glb_nsz = 16
import re
import os
from PySide.QtCore import *
@@ -102,6 +111,15 @@ from decimal import *
from ctypes import *
from multiprocessing import Process, Array, Value, Event
+# xrange is range in Python3
+try:
+ xrange
+except NameError:
+ xrange = range
+
+def printerr(*args, **keyword_args):
+ print(*args, file=sys.stderr, **keyword_args)
+
# Data formatting helpers
def tohex(ip):
@@ -1004,10 +1022,6 @@ class ChildDataItemFinder():
glb_chunk_sz = 1
-# size of pickled integer big enough for record size
-
-glb_nsz = 8
-
# Background process for SQL data fetcher
class SQLFetcherProcess():
@@ -1066,7 +1080,7 @@ class SQLFetcherProcess():
return True
if space >= glb_nsz:
# Use 0 (or space < glb_nsz) to mean there is
no more at the top of the buffer
- nd = cPickle.dumps(0, cPickle.HIGHEST_PROTOCOL)
+ nd = pickle.dumps(0, pickle.HIGHEST_PROTOCOL)
self.buffer[self.local_head : self.local_head +
len(nd)] = nd
self.local_head = 0
if self.local_tail - self.local_head > sz:
@@ -1084,9 +1098,9 @@ class SQLFetcherProcess():
self.wait_event.wait()
def AddToBuffer(self, obj):
- d = cPickle.dumps(obj, cPickle.HIGHEST_PROTOCOL)
+ d = pickle.dumps(obj, pickle.HIGHEST_PROTOCOL)
n = len(d)
- nd = cPickle.dumps(n, cPickle.HIGHEST_PROTOCOL)
+ nd = pickle.dumps(n, pickle.HIGHEST_PROTOCOL)
sz = n + glb_nsz
self.WaitForSpace(sz)
pos = self.local_head
@@ -1198,12 +1212,12 @@ class SQLFetcher(QObject):
pos = self.local_tail
if len(self.buffer) - pos < glb_nsz:
pos = 0
- n = cPickle.loads(self.buffer[pos : pos + glb_nsz])
+ n = pickle.loads(self.buffer[pos : pos + glb_nsz])
if n == 0:
pos = 0
- n = cPickle.loads(self.buffer[0 : glb_nsz])
+ n = pickle.loads(self.buffer[0 : glb_nsz])
pos += glb_nsz
- obj = cPickle.loads(self.buffer[pos : pos + n])
+ obj = pickle.loads(self.buffer[pos : pos + n])
self.local_tail = pos + n
return obj
@@ -2973,7 +2987,7 @@ class DBRef():
def Main():
if (len(sys.argv) < 2):
- print >> sys.stderr, "Usage is: exported-sql-viewer.py
{ | --help-only}"
+ printerr("Usage is: exported-sql-viewer.py { |
--help-only}");
raise Exception("Too few arguments")
dbname = sys.argv[1]
@@ -2986,8 +3000,8 @@ def Main():
is_sqlite3 = False
try:
- f = open(dbname)
- if f.read(15) == "SQLite format 3":
+ f = open(dbname, "rb")
+ if f.read(15) == b'SQLite format 3':
is_sqlite3 = True
f.close()
except:
[tip:perf/urgent] perf script python: Add printdate function to SQL exporters
Commit-ID: 49f93bbf17e6267eb34e0c12a9813f3a8723749e
Gitweb: https://git.kernel.org/tip/49f93bbf17e6267eb34e0c12a9813f3a8723749e
Author: Tony Jones
AuthorDate: Fri, 8 Mar 2019 16:05:18 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Mon, 11 Mar 2019 16:13:02 -0300
perf script python: Add printdate function to SQL exporters
Introduce a printdate function to eliminate the repetitive use of
datetime.datetime.today() in the SQL exporting scripts.
Signed-off-by: Tony Jones
Acked-by: Adrian Hunter
Link: http://lkml.kernel.org/r/20190309000518.2438-5-to...@suse.de
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/export-to-postgresql.py | 19 +++
tools/perf/scripts/python/export-to-sqlite.py | 13 -
2 files changed, 19 insertions(+), 13 deletions(-)
diff --git a/tools/perf/scripts/python/export-to-postgresql.py
b/tools/perf/scripts/python/export-to-postgresql.py
index 00ab972a2eba..c3eae1d77d36 100644
--- a/tools/perf/scripts/python/export-to-postgresql.py
+++ b/tools/perf/scripts/python/export-to-postgresql.py
@@ -251,6 +251,9 @@ perf_db_export_callchains = False
def printerr(*args, **kw_args):
print(*args, file=sys.stderr, **kw_args)
+def printdate(*args, **kw_args):
+print(datetime.datetime.today(), *args, sep=' ', **kw_args)
+
def usage():
printerr("Usage is: export-to-postgresql.py []
[] []")
printerr("where:columns 'all' or 'branches'")
@@ -289,7 +292,7 @@ def do_query(q, s):
return
raise Exception("Query failed: " + q.lastError().text())
-print(datetime.datetime.today(), "Creating database...")
+printdate("Creating database...")
db = QSqlDatabase.addDatabase('QPSQL')
query = QSqlQuery(db)
@@ -582,7 +585,7 @@ if perf_db_export_calls:
call_file = open_output_file("call_table.bin")
def trace_begin():
- print(datetime.datetime.today(), "Writing to intermediate files...")
+ printdate("Writing to intermediate files...")
# id == 0 means unknown. It is easier to create records for them than
replace the zeroes with NULLs
evsel_table(0, "unknown")
machine_table(0, 0, "unknown")
@@ -598,7 +601,7 @@ def trace_begin():
unhandled_count = 0
def trace_end():
- print(datetime.datetime.today(), "Copying to database...")
+ printdate("Copying to database...")
copy_output_file(evsel_file,"selected_events")
copy_output_file(machine_file, "machines")
copy_output_file(thread_file, "threads")
@@ -613,7 +616,7 @@ def trace_end():
if perf_db_export_calls:
copy_output_file(call_file, "calls")
- print(datetime.datetime.today(), "Removing intermediate files...")
+ printdate("Removing intermediate files...")
remove_output_file(evsel_file)
remove_output_file(machine_file)
remove_output_file(thread_file)
@@ -628,7 +631,7 @@ def trace_end():
if perf_db_export_calls:
remove_output_file(call_file)
os.rmdir(output_dir_name)
- print(datetime.datetime.today(), "Adding primary keys")
+ printdate("Adding primary keys")
do_query(query, 'ALTER TABLE selected_events ADD PRIMARY KEY (id)')
do_query(query, 'ALTER TABLE machinesADD PRIMARY KEY (id)')
do_query(query, 'ALTER TABLE threads ADD PRIMARY KEY (id)')
@@ -643,7 +646,7 @@ def trace_end():
if perf_db_export_calls:
do_query(query, 'ALTER TABLE calls ADD PRIMARY KEY
(id)')
- print(datetime.datetime.today(), "Adding foreign keys")
+ printdate("Adding foreign keys")
do_query(query, 'ALTER TABLE threads '
'ADD CONSTRAINT machinefk FOREIGN KEY
(machine_id) REFERENCES machines (id),'
'ADD CONSTRAINT processfk FOREIGN KEY
(process_id) REFERENCES threads(id)')
@@ -679,8 +682,8 @@ def trace_end():
do_query(query, 'CREATE INDEX pid_idx ON calls (parent_id)')
if (unhandled_count):
- print(datetime.datetime.today(), "Warning: ", unhandled_count,
" unhandled events")
- print(datetime.datetime.today(), "Done")
+ printdate("Warning: ", unhandled_count, " unhandled events")
+ printdate("Done")
def trace_unhandled(event_name, context, event_fields_dict):
global unhandled_count
diff --git a/tools/perf/scripts/python/export-to-sqlite.py
b/tools/perf/scripts/python/export-to-sqlite.py
index 3da338243aed..3b71902a5a21 100644
--- a/tools/perf/scripts/python/export-to-
[tip:perf/urgent] perf script python: Add Python3 support to export-to-postgresql.py
Commit-ID: 1937b0560c3ea43b1b0f7d3617949ca50de8f8c0
Gitweb: https://git.kernel.org/tip/1937b0560c3ea43b1b0f7d3617949ca50de8f8c0
Author: Tony Jones
AuthorDate: Fri, 8 Mar 2019 16:05:16 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Mon, 11 Mar 2019 16:12:57 -0300
perf script python: Add Python3 support to export-to-postgresql.py
Support both Python2 and Python3 in the export-to-postgresql.py script.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Link: http://lkml.kernel.org/r/20190309000518.2438-3-to...@suse.de
Signed-off-by: Adrian Hunter
Signed-off-by: Seeteena Thoufeek
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/export-to-postgresql.py | 58 ---
1 file changed, 41 insertions(+), 17 deletions(-)
diff --git a/tools/perf/scripts/python/export-to-postgresql.py
b/tools/perf/scripts/python/export-to-postgresql.py
index 390a351d15ea..00ab972a2eba 100644
--- a/tools/perf/scripts/python/export-to-postgresql.py
+++ b/tools/perf/scripts/python/export-to-postgresql.py
@@ -10,6 +10,8 @@
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
+from __future__ import print_function
+
import os
import sys
import struct
@@ -199,6 +201,18 @@ import datetime
from PySide.QtSql import *
+if sys.version_info < (3, 0):
+ def toserverstr(str):
+ return str
+ def toclientstr(str):
+ return str
+else:
+ # Assume UTF-8 server_encoding and client_encoding
+ def toserverstr(str):
+ return bytes(str, "UTF_8")
+ def toclientstr(str):
+ return bytes(str, "UTF_8")
+
# Need to access PostgreSQL C library directly to use COPY FROM STDIN
from ctypes import *
libpq = CDLL("libpq.so.5")
@@ -234,12 +248,14 @@ perf_db_export_mode = True
perf_db_export_calls = False
perf_db_export_callchains = False
+def printerr(*args, **kw_args):
+ print(*args, file=sys.stderr, **kw_args)
def usage():
- print >> sys.stderr, "Usage is: export-to-postgresql.py
[] [] []"
- print >> sys.stderr, "where:columns 'all' or 'branches'"
- print >> sys.stderr, " calls 'calls' => create calls
and call_paths table"
- print >> sys.stderr, " callchains 'callchains' => create
call_paths table"
+ printerr("Usage is: export-to-postgresql.py []
[] []")
+ printerr("where:columns 'all' or 'branches'")
+ printerr(" calls 'calls' => create calls and
call_paths table")
+ printerr(" callchains 'callchains' => create
call_paths table")
raise Exception("Too few arguments")
if (len(sys.argv) < 2):
@@ -273,7 +289,7 @@ def do_query(q, s):
return
raise Exception("Query failed: " + q.lastError().text())
-print datetime.datetime.today(), "Creating database..."
+print(datetime.datetime.today(), "Creating database...")
db = QSqlDatabase.addDatabase('QPSQL')
query = QSqlQuery(db)
@@ -506,12 +522,12 @@ do_query(query, 'CREATE VIEW samples_view AS '
' FROM samples')
-file_header = struct.pack("!11sii", "PGCOPY\n\377\r\n\0", 0, 0)
-file_trailer = "\377\377"
+file_header = struct.pack("!11sii", b"PGCOPY\n\377\r\n\0", 0, 0)
+file_trailer = b"\377\377"
def open_output_file(file_name):
path_name = output_dir_name + "/" + file_name
- file = open(path_name, "w+")
+ file = open(path_name, "wb+")
file.write(file_header)
return file
@@ -526,13 +542,13 @@ def copy_output_file_direct(file, table_name):
# Use COPY FROM STDIN because security may prevent postgres from accessing the
files directly
def copy_output_file(file, table_name):
- conn = PQconnectdb("dbname = " + dbname)
+ conn = PQconnectdb(toclientstr("dbname = " + dbname))
if (PQstatus(conn)):
raise Exception("COPY FROM STDIN PQconnectdb failed")
file.write(file_trailer)
file.seek(0)
sql = "COPY " + table_name + " FROM STDIN (FORMAT 'binary')"
- res = PQexec(conn, sql)
+ res = PQexec(conn, toclientstr(sql))
if (PQresultStatus(res) != 4):
raise Exception("COPY FROM STDIN PQexec failed")
data = file.read(65536)
@@ -566,7 +582,7 @@ if perf_db_export_calls:
call_file = open_output_file("call_table.bin")
def trace_begin():
- print datetime.datetime.today(), "Writing to intermediate files..."
+ print(datetime.date
Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption
On 3/21/19 11:20 AM, Peter Zijlstra wrote: > Might be best to have someone test this.. someone with ucode and a > machine etc.. :-) If your own company can't oblige you here Peter :-) I'm sure I can make some time to test it.
[tip:perf/urgent] perf script python: Add Python3 support to intel-pt-events.py
Commit-ID: fdf2460c297f1bb2f3bd20b3b52903b267af9050
Gitweb: https://git.kernel.org/tip/fdf2460c297f1bb2f3bd20b3b52903b267af9050
Author: Tony Jones
AuthorDate: Tue, 5 Mar 2019 08:19:02 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Wed, 6 Mar 2019 18:12:33 -0300
perf script python: Add Python3 support to intel-pt-events.py
Support both Python2 and Python3 in the intel-pt-events.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Acked-by: Adrian Hunter
Link: http://lkml.kernel.org/r/fd26acf9-0c0f-717f-9664-a3c33043c...@suse.de
Signed-off-by: Seeteena Thoufeek
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/intel-pt-events.py | 32 +---
1 file changed, 19 insertions(+), 13 deletions(-)
diff --git a/tools/perf/scripts/python/intel-pt-events.py
b/tools/perf/scripts/python/intel-pt-events.py
index 2177722f509e..a73847c8f548 100644
--- a/tools/perf/scripts/python/intel-pt-events.py
+++ b/tools/perf/scripts/python/intel-pt-events.py
@@ -10,6 +10,8 @@
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
+from __future__ import print_function
+
import os
import sys
import struct
@@ -22,34 +24,34 @@ sys.path.append(os.environ['PERF_EXEC_PATH'] + \
#from Core import *
def trace_begin():
- print "Intel PT Power Events and PTWRITE"
+ print("Intel PT Power Events and PTWRITE")
def trace_end():
- print "End"
+ print("End")
def trace_unhandled(event_name, context, event_fields_dict):
- print ' '.join(['%s=%s'%(k,str(v))for k,v in
sorted(event_fields_dict.items())])
+ print(' '.join(['%s=%s'%(k,str(v))for k,v in
sorted(event_fields_dict.items())]))
def print_ptwrite(raw_buf):
data = struct.unpack_from("> 32) & 0x3
- print "hints: %#x extensions: %#x" % (hints, extensions),
+ print("hints: %#x extensions: %#x" % (hints, extensions), end=' ')
def print_pwre(raw_buf):
data = struct.unpack_from("> 7) & 1
cstate = (payload >> 12) & 0xf
subcstate = (payload >> 8) & 0xf
- print "hw: %u cstate: %u sub-cstate: %u" % (hw, cstate, subcstate),
+ print("hw: %u cstate: %u sub-cstate: %u" % (hw, cstate, subcstate),
+ end=' ')
def print_exstop(raw_buf):
data = struct.unpack_from("> 4) & 0xf
wake_reason = (payload >> 8) & 0xf
- print "deepest cstate: %u last cstate: %u wake reason: %#x" %
(deepest_cstate, last_cstate, wake_reason),
+ print("deepest cstate: %u last cstate: %u wake reason: %#x" %
+ (deepest_cstate, last_cstate, wake_reason), end=' ')
def print_common_start(comm, sample, name):
ts = sample["time"]
cpu = sample["cpu"]
pid = sample["pid"]
tid = sample["tid"]
- print "%16s %5u/%-5u [%03u] %9u.%09u %7s:" % (comm, pid, tid, cpu, ts /
10, ts %10, name),
+ print("%16s %5u/%-5u [%03u] %9u.%09u %7s:" %
+ (comm, pid, tid, cpu, ts / 10, ts %10, name),
+ end=' ')
def print_common_ip(sample, symbol, dso):
ip = sample["ip"]
- print "%16x %s (%s)" % (ip, symbol, dso)
+ print("%16x %s (%s)" % (ip, symbol, dso))
def process_event(param_dict):
event_attr = param_dict["attr"]
@@ -92,12 +98,12 @@ def process_event(param_dict):
name = param_dict["ev_name"]
# Symbol and dso info are not always resolved
- if (param_dict.has_key("dso")):
+ if "dso" in param_dict:
dso = param_dict["dso"]
else:
dso = "[unknown]"
- if (param_dict.has_key("symbol")):
+ if "symbol" in param_dict:
symbol = param_dict["symbol"]
else:
symbol = "[unknown]"
[tip:perf/urgent] perf script python: add Python3 support to check-perf-trace.py
Commit-ID: 57e604b16362273af6a517abaa6cd1133a7fc732
Gitweb: https://git.kernel.org/tip/57e604b16362273af6a517abaa6cd1133a7fc732
Author: Tony Jones
AuthorDate: Fri, 1 Mar 2019 17:18:59 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Wed, 6 Mar 2019 18:10:46 -0300
perf script python: add Python3 support to check-perf-trace.py
Support both Python 2 and Python 3 in the check-perf-trace.py script.
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of from __future__ implies the minimum supported version of
Python2 is now v2.6
Signed-off-by: Tony Jones
Cc: Tom Zanussi
Link: http://lkml.kernel.org/r/20190302011903.2416-4-to...@suse.de
Signed-off-by: Seeteena Thoufeek
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/check-perf-trace.py | 31 +++
1 file changed, 17 insertions(+), 14 deletions(-)
diff --git a/tools/perf/scripts/python/check-perf-trace.py
b/tools/perf/scripts/python/check-perf-trace.py
index f4838db3e518..d2c22954800d 100644
--- a/tools/perf/scripts/python/check-perf-trace.py
+++ b/tools/perf/scripts/python/check-perf-trace.py
@@ -7,6 +7,8 @@
# events, etc. Basically, if this script runs successfully and
# displays expected results, Python scripting support should be ok.
+from __future__ import print_function
+
import os
import sys
@@ -19,7 +21,7 @@ from perf_trace_context import *
unhandled = autodict()
def trace_begin():
- print "trace_begin"
+ print("trace_begin")
pass
def trace_end():
@@ -33,7 +35,7 @@ def irq__softirq_entry(event_name, context, common_cpu,
print_uncommon(context)
- print "vec=%s\n" % (symbol_str("irq__softirq_entry", "vec", vec)),
+ print("vec=%s" % (symbol_str("irq__softirq_entry", "vec", vec)))
def kmem__kmalloc(event_name, context, common_cpu,
common_secs, common_nsecs, common_pid, common_comm,
@@ -44,10 +46,10 @@ def kmem__kmalloc(event_name, context, common_cpu,
print_uncommon(context)
- print "call_site=%u, ptr=%u, bytes_req=%u, " \
- "bytes_alloc=%u, gfp_flags=%s\n" % \
+ print("call_site=%u, ptr=%u, bytes_req=%u, "
+ "bytes_alloc=%u, gfp_flags=%s" %
(call_site, ptr, bytes_req, bytes_alloc,
- flag_str("kmem__kmalloc", "gfp_flags", gfp_flags)),
+ flag_str("kmem__kmalloc", "gfp_flags", gfp_flags)))
def trace_unhandled(event_name, context, event_fields_dict):
try:
@@ -56,26 +58,27 @@ def trace_unhandled(event_name, context, event_fields_dict):
unhandled[event_name] = 1
def print_header(event_name, cpu, secs, nsecs, pid, comm):
- print "%-20s %5u %05u.%09u %8u %-20s " % \
+ print("%-20s %5u %05u.%09u %8u %-20s " %
(event_name, cpu, secs, nsecs, pid, comm),
+ end=' ')
# print trace fields not included in handler args
def print_uncommon(context):
- print "common_preempt_count=%d, common_flags=%s, " \
- "common_lock_depth=%d, " % \
+ print("common_preempt_count=%d, common_flags=%s, "
+ "common_lock_depth=%d, " %
(common_pc(context), trace_flag_str(common_flags(context)),
- common_lock_depth(context))
+ common_lock_depth(context)))
def print_unhandled():
keys = unhandled.keys()
if not keys:
return
- print "\nunhandled events:\n\n",
+ print("\nunhandled events:\n")
- print "%-40s %10s\n" % ("event", "count"),
- print "%-40s %10s\n" % ("", \
- "---"),
+ print("%-40s %10s" % ("event", "count"))
+ print("%-40s %10s" % ("",
+ "---"))
for event_name in keys:
- print "%-40s %10d\n" % (event_name, unhandled[event_name])
+ print("%-40s %10d\n" % (event_name, unhandled[event_name]))
[tip:perf/urgent] perf script python: Add Python3 support to event_analyzing_sample.py
Commit-ID: c253c72e9d6723c8b078beb362f050059ef5de39
Gitweb: https://git.kernel.org/tip/c253c72e9d6723c8b078beb362f050059ef5de39
Author: Tony Jones
AuthorDate: Fri, 1 Mar 2019 17:19:00 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Wed, 6 Mar 2019 18:11:11 -0300
perf script python: Add Python3 support to event_analyzing_sample.py
Support both Python2 and Python3 in the event_analyzing_sample.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Cc: Feng Tang
Link: http://lkml.kernel.org/r/20190302011903.2416-5-to...@suse.de
Signed-off-by: Seeteena Thoufeek
Signed-off-by: Arnaldo Carvalho de Melo
---
.../perf/scripts/python/event_analyzing_sample.py | 48 +++---
1 file changed, 25 insertions(+), 23 deletions(-)
diff --git a/tools/perf/scripts/python/event_analyzing_sample.py
b/tools/perf/scripts/python/event_analyzing_sample.py
index 2ec8915b74c5..aa1e2cfa26a6 100644
--- a/tools/perf/scripts/python/event_analyzing_sample.py
+++ b/tools/perf/scripts/python/event_analyzing_sample.py
@@ -15,6 +15,8 @@
# for a x86 HW PMU event: PEBS with load latency data.
#
+from __future__ import print_function
+
import os
import sys
import math
@@ -37,7 +39,7 @@ con = sqlite3.connect("/dev/shm/perf.db")
con.isolation_level = None
def trace_begin():
-print "In trace_begin:\n"
+print("In trace_begin:\n")
#
# Will create several tables at the start, pebs_ll is for PEBS data
with
@@ -76,12 +78,12 @@ def process_event(param_dict):
name = param_dict["ev_name"]
# Symbol and dso info are not always resolved
-if (param_dict.has_key("dso")):
+if ("dso" in param_dict):
dso = param_dict["dso"]
else:
dso = "Unknown_dso"
-if (param_dict.has_key("symbol")):
+if ("symbol" in param_dict):
symbol = param_dict["symbol"]
else:
symbol = "Unknown_symbol"
@@ -102,7 +104,7 @@ def insert_db(event):
event.ip, event.status, event.dse, event.dla,
event.lat))
def trace_end():
-print "In trace_end:\n"
+print("In trace_end:\n")
# We show the basic info for the 2 type of event classes
show_general_events()
show_pebs_ll()
@@ -123,29 +125,29 @@ def show_general_events():
# Check the total record number in the table
count = con.execute("select count(*) from gen_events")
for t in count:
-print "There is %d records in gen_events table" % t[0]
+print("There is %d records in gen_events table" % t[0])
if t[0] == 0:
return
-print "Statistics about the general events grouped by
thread/symbol/dso: \n"
+print("Statistics about the general events grouped by
thread/symbol/dso: \n")
# Group by thread
commq = con.execute("select comm, count(comm) from gen_events group by
comm order by -count(comm)")
-print "\n%16s %8s %16s\n%s" % ("comm", "number", "histogram", "="*42)
+print("\n%16s %8s %16s\n%s" % ("comm", "number", "histogram", "="*42))
for row in commq:
- print "%16s %8d %s" % (row[0], row[1], num2sym(row[1]))
+ print("%16s %8d %s" % (row[0], row[1], num2sym(row[1])))
# Group by symbol
-print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histogram", "="*58)
+print("\n%32s %8s %16s\n%s" % ("symbol", "number", "histogram",
"="*58))
symbolq = con.execute("select symbol, count(symbol) from gen_events
group by symbol order by -count(symbol)")
for row in symbolq:
- print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))
+ print("%32s %8d %s" % (row[0], row[1], num2sym(row[1])))
# Group by dso
-print "\n%40s %8s %16s\n%s" % ("dso", "number", "histogram", "="*74)
+print("\n%40s %8s %16s\n%s" % ("dso", "number", "histogram", "="*74))
dsoq = con.execute("select dso, count(dso) from gen_events group by
dso order by -count(dso)")
for row in dsoq:
- pr
[tip:perf/urgent] perf script python: Add Python3 support to futex-contention.py
Commit-ID: de2ec16bd438945813198d4de2339a396904c206
Gitweb: https://git.kernel.org/tip/de2ec16bd438945813198d4de2339a396904c206
Author: Tony Jones
AuthorDate: Fri, 1 Mar 2019 17:18:58 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Wed, 6 Mar 2019 18:10:43 -0300
perf script python: Add Python3 support to futex-contention.py
Support both Python2 and Python3 in the futex-contention.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Link: http://lkml.kernel.org/r/20190302011903.2416-3-to...@suse.de
Signed-off-by: Seeteena Thoufeek
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/futex-contention.py | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/tools/perf/scripts/python/futex-contention.py
b/tools/perf/scripts/python/futex-contention.py
index f221c62e0a10..0c4841acf75d 100644
--- a/tools/perf/scripts/python/futex-contention.py
+++ b/tools/perf/scripts/python/futex-contention.py
@@ -10,6 +10,8 @@
#
# Measures futex contention
+from __future__ import print_function
+
import os, sys
sys.path.append(os.environ['PERF_EXEC_PATH'] +
'/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
from Util import *
@@ -33,18 +35,18 @@ def syscalls__sys_enter_futex(event, ctxt, cpu, s, ns, tid,
comm, callchain,
def syscalls__sys_exit_futex(event, ctxt, cpu, s, ns, tid, comm, callchain,
nr, ret):
- if thread_blocktime.has_key(tid):
+ if tid in thread_blocktime:
elapsed = nsecs(s, ns) - thread_blocktime[tid]
add_stats(lock_waits, (tid, thread_thislock[tid]), elapsed)
del thread_blocktime[tid]
del thread_thislock[tid]
def trace_begin():
- print "Press control+C to stop and show the summary"
+ print("Press control+C to stop and show the summary")
def trace_end():
for (tid, lock) in lock_waits:
min, max, avg, count = lock_waits[tid, lock]
- print "%s[%d] lock %x contended %d times, %d avg ns" % \
- (process_names[tid], tid, lock, count, avg)
+ print("%s[%d] lock %x contended %d times, %d avg ns" %
+ (process_names[tid], tid, lock, count, avg))
[tip:perf/urgent] perf script python: Remove mixed indentation
Commit-ID: b504d7f6876515b74c8e27a44ccdb22372616d97
Gitweb: https://git.kernel.org/tip/b504d7f6876515b74c8e27a44ccdb22372616d97
Author: Tony Jones
AuthorDate: Fri, 1 Mar 2019 17:18:57 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Wed, 6 Mar 2019 18:09:14 -0300
perf script python: Remove mixed indentation
Remove mixed indentation in Python scripts. Revert to either all tabs
(most common form) or all spaces (4 or 8) depending on what was the
intent of the original commit. This is necessary to complete Python3
support as it will flag an error if it encounters mixed indentation.
Signed-off-by: Tony Jones
Link: http://lkml.kernel.org/r/20190302011903.2416-2-to...@suse.de
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/check-perf-trace.py | 65 +++---
tools/perf/scripts/python/compaction-times.py | 8 +--
.../perf/scripts/python/event_analyzing_sample.py | 6 +-
.../perf/scripts/python/failed-syscalls-by-pid.py | 38 ++---
tools/perf/scripts/python/futex-contention.py | 2 +-
tools/perf/scripts/python/intel-pt-events.py | 32 +--
tools/perf/scripts/python/mem-phys-addr.py | 7 ++-
tools/perf/scripts/python/net_dropmonitor.py | 2 +-
tools/perf/scripts/python/netdev-times.py | 12 ++--
tools/perf/scripts/python/sched-migration.py | 6 +-
tools/perf/scripts/python/sctop.py | 13 +++--
tools/perf/scripts/python/stackcollapse.py | 2 +-
tools/perf/scripts/python/syscall-counts-by-pid.py | 47
tools/perf/scripts/python/syscall-counts.py| 31 +--
14 files changed, 136 insertions(+), 135 deletions(-)
diff --git a/tools/perf/scripts/python/check-perf-trace.py
b/tools/perf/scripts/python/check-perf-trace.py
index 334599c6032c..f4838db3e518 100644
--- a/tools/perf/scripts/python/check-perf-trace.py
+++ b/tools/perf/scripts/python/check-perf-trace.py
@@ -23,60 +23,59 @@ def trace_begin():
pass
def trace_end():
-print_unhandled()
+ print_unhandled()
def irq__softirq_entry(event_name, context, common_cpu,
- common_secs, common_nsecs, common_pid, common_comm,
- common_callchain, vec):
- print_header(event_name, common_cpu, common_secs, common_nsecs,
- common_pid, common_comm)
+ common_secs, common_nsecs, common_pid, common_comm,
+ common_callchain, vec):
+ print_header(event_name, common_cpu, common_secs, common_nsecs,
+ common_pid, common_comm)
-print_uncommon(context)
+ print_uncommon(context)
- print "vec=%s\n" % \
- (symbol_str("irq__softirq_entry", "vec", vec)),
+ print "vec=%s\n" % (symbol_str("irq__softirq_entry", "vec", vec)),
def kmem__kmalloc(event_name, context, common_cpu,
- common_secs, common_nsecs, common_pid, common_comm,
- common_callchain, call_site, ptr, bytes_req, bytes_alloc,
- gfp_flags):
- print_header(event_name, common_cpu, common_secs, common_nsecs,
- common_pid, common_comm)
+ common_secs, common_nsecs, common_pid, common_comm,
+ common_callchain, call_site, ptr, bytes_req, bytes_alloc,
+ gfp_flags):
+ print_header(event_name, common_cpu, common_secs, common_nsecs,
+ common_pid, common_comm)
-print_uncommon(context)
+ print_uncommon(context)
- print "call_site=%u, ptr=%u, bytes_req=%u, " \
+ print "call_site=%u, ptr=%u, bytes_req=%u, " \
"bytes_alloc=%u, gfp_flags=%s\n" % \
(call_site, ptr, bytes_req, bytes_alloc,
-
flag_str("kmem__kmalloc", "gfp_flags", gfp_flags)),
def trace_unhandled(event_name, context, event_fields_dict):
-try:
-unhandled[event_name] += 1
-except TypeError:
-unhandled[event_name] = 1
+ try:
+ unhandled[event_name] += 1
+ except TypeError:
+ unhandled[event_name] = 1
def print_header(event_name, cpu, secs, nsecs, pid, comm):
print "%-20s %5u %05u.%09u %8u %-20s " % \
- (event_name, cpu, secs, nsecs, pid, comm),
+ (event_name, cpu, secs, nsecs, pid, comm),
# print trace fields not included in handler args
def print_uncommon(context):
-print "common_preempt_count=%d, common_flags=%s, common_lock_depth=%d, " \
-% (common_pc(context), trace_flag_str(common_flags(context)), \
- common_lock_depth(context))
+ print "common_preempt_count=%d, common_flags=%s, " \
+ "common_lock_depth=%d, " % \
+ (common_pc(context), trace_flag_str(common_flags(context)),
+
[tip:perf/urgent] tools lib traceevent: Fix buffer overflow in arg_eval
Commit-ID: 7c5b019e3a638a5a290b0ec020f6ca83d2ec2aaa
Gitweb: https://git.kernel.org/tip/7c5b019e3a638a5a290b0ec020f6ca83d2ec2aaa
Author: Tony Jones
AuthorDate: Wed, 27 Feb 2019 17:55:32 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Thu, 28 Feb 2019 16:06:47 -0300
tools lib traceevent: Fix buffer overflow in arg_eval
Fix buffer overflow observed when running perf test.
The overflow is when trying to evaluate "1ULL << (64 - 1)" which is
resulting in -9223372036854775808 which overflows the 20 character
buffer.
If is possible this bug has been reported before but I still don't see
any fix checked in:
See: https://www.spinics.net/lists/linux-perf-users/msg07714.html
Reported-by: Michael Sartain
Reported-by: Mathias Krause
Signed-off-by: Tony Jones
Acked-by: Steven Rostedt (VMware)
Cc: Frederic Weisbecker
Fixes: f7d82350e597 ("tools/events: Add files to create libtraceevent.a")
Link: http://lkml.kernel.org/r/20190228015532.8941-1-to...@suse.de
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/lib/traceevent/event-parse.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/lib/traceevent/event-parse.c
b/tools/lib/traceevent/event-parse.c
index abd4fa5d3088..87494c7c619d 100644
--- a/tools/lib/traceevent/event-parse.c
+++ b/tools/lib/traceevent/event-parse.c
@@ -2457,7 +2457,7 @@ static int arg_num_eval(struct tep_print_arg *arg, long
long *val)
static char *arg_eval (struct tep_print_arg *arg)
{
long long val;
- static char buf[20];
+ static char buf[24];
switch (arg->type) {
case TEP_PRINT_ATOM:
[PATCH v3 4/4] perf script python: add printdate function to SQL exporters
Introduce a printdate function to eliminate the repetitive use of
datetime.datetime.today() in the SQL exporting scripts.
Signed-off-by: Tony Jones
Acked-by: Adrian Hunter
---
tools/perf/scripts/python/export-to-postgresql.py | 19 +++
tools/perf/scripts/python/export-to-sqlite.py | 13 -
2 files changed, 19 insertions(+), 13 deletions(-)
diff --git a/tools/perf/scripts/python/export-to-postgresql.py
b/tools/perf/scripts/python/export-to-postgresql.py
index 00ab972a2eba..c3eae1d77d36 100644
--- a/tools/perf/scripts/python/export-to-postgresql.py
+++ b/tools/perf/scripts/python/export-to-postgresql.py
@@ -251,6 +251,9 @@ perf_db_export_callchains = False
def printerr(*args, **kw_args):
print(*args, file=sys.stderr, **kw_args)
+def printdate(*args, **kw_args):
+print(datetime.datetime.today(), *args, sep=' ', **kw_args)
+
def usage():
printerr("Usage is: export-to-postgresql.py []
[] []")
printerr("where:columns 'all' or 'branches'")
@@ -289,7 +292,7 @@ def do_query(q, s):
return
raise Exception("Query failed: " + q.lastError().text())
-print(datetime.datetime.today(), "Creating database...")
+printdate("Creating database...")
db = QSqlDatabase.addDatabase('QPSQL')
query = QSqlQuery(db)
@@ -582,7 +585,7 @@ if perf_db_export_calls:
call_file = open_output_file("call_table.bin")
def trace_begin():
- print(datetime.datetime.today(), "Writing to intermediate files...")
+ printdate("Writing to intermediate files...")
# id == 0 means unknown. It is easier to create records for them than
replace the zeroes with NULLs
evsel_table(0, "unknown")
machine_table(0, 0, "unknown")
@@ -598,7 +601,7 @@ def trace_begin():
unhandled_count = 0
def trace_end():
- print(datetime.datetime.today(), "Copying to database...")
+ printdate("Copying to database...")
copy_output_file(evsel_file,"selected_events")
copy_output_file(machine_file, "machines")
copy_output_file(thread_file, "threads")
@@ -613,7 +616,7 @@ def trace_end():
if perf_db_export_calls:
copy_output_file(call_file, "calls")
- print(datetime.datetime.today(), "Removing intermediate files...")
+ printdate("Removing intermediate files...")
remove_output_file(evsel_file)
remove_output_file(machine_file)
remove_output_file(thread_file)
@@ -628,7 +631,7 @@ def trace_end():
if perf_db_export_calls:
remove_output_file(call_file)
os.rmdir(output_dir_name)
- print(datetime.datetime.today(), "Adding primary keys")
+ printdate("Adding primary keys")
do_query(query, 'ALTER TABLE selected_events ADD PRIMARY KEY (id)')
do_query(query, 'ALTER TABLE machinesADD PRIMARY KEY (id)')
do_query(query, 'ALTER TABLE threads ADD PRIMARY KEY (id)')
@@ -643,7 +646,7 @@ def trace_end():
if perf_db_export_calls:
do_query(query, 'ALTER TABLE calls ADD PRIMARY KEY
(id)')
- print(datetime.datetime.today(), "Adding foreign keys")
+ printdate("Adding foreign keys")
do_query(query, 'ALTER TABLE threads '
'ADD CONSTRAINT machinefk FOREIGN KEY
(machine_id) REFERENCES machines (id),'
'ADD CONSTRAINT processfk FOREIGN KEY
(process_id) REFERENCES threads(id)')
@@ -679,8 +682,8 @@ def trace_end():
do_query(query, 'CREATE INDEX pid_idx ON calls (parent_id)')
if (unhandled_count):
- print(datetime.datetime.today(), "Warning: ", unhandled_count,
" unhandled events")
- print(datetime.datetime.today(), "Done")
+ printdate("Warning: ", unhandled_count, " unhandled events")
+ printdate("Done")
def trace_unhandled(event_name, context, event_fields_dict):
global unhandled_count
diff --git a/tools/perf/scripts/python/export-to-sqlite.py
b/tools/perf/scripts/python/export-to-sqlite.py
index 3da338243aed..3b71902a5a21 100644
--- a/tools/perf/scripts/python/export-to-sqlite.py
+++ b/tools/perf/scripts/python/export-to-sqlite.py
@@ -65,6 +65,9 @@ perf_db_export_callchains = False
def printerr(*args, **keyword_args):
print(*args, file=sys.stderr, **keyword_args)
+def printdate(*args, **kw_args):
+print(datetime.datetime.today(), *args, sep=' ', **kw_args)
+
def usage():
printerr("Usage is: export-to-sqlite.py []
[] []");
printerr("where:columns
[PATCH v3 2/4] perf script python: add Python3 support to export-to-postgresql.py
Support both Python2 and Python3 in the export-to-postgresql.py script.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Adrian Hunter
Signed-off-by: Seeteena Thoufeek
---
tools/perf/scripts/python/export-to-postgresql.py | 58 ---
1 file changed, 41 insertions(+), 17 deletions(-)
diff --git a/tools/perf/scripts/python/export-to-postgresql.py
b/tools/perf/scripts/python/export-to-postgresql.py
index 390a351d15ea..00ab972a2eba 100644
--- a/tools/perf/scripts/python/export-to-postgresql.py
+++ b/tools/perf/scripts/python/export-to-postgresql.py
@@ -10,6 +10,8 @@
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
+from __future__ import print_function
+
import os
import sys
import struct
@@ -199,6 +201,18 @@ import datetime
from PySide.QtSql import *
+if sys.version_info < (3, 0):
+ def toserverstr(str):
+ return str
+ def toclientstr(str):
+ return str
+else:
+ # Assume UTF-8 server_encoding and client_encoding
+ def toserverstr(str):
+ return bytes(str, "UTF_8")
+ def toclientstr(str):
+ return bytes(str, "UTF_8")
+
# Need to access PostgreSQL C library directly to use COPY FROM STDIN
from ctypes import *
libpq = CDLL("libpq.so.5")
@@ -234,12 +248,14 @@ perf_db_export_mode = True
perf_db_export_calls = False
perf_db_export_callchains = False
+def printerr(*args, **kw_args):
+ print(*args, file=sys.stderr, **kw_args)
def usage():
- print >> sys.stderr, "Usage is: export-to-postgresql.py
[] [] []"
- print >> sys.stderr, "where:columns 'all' or 'branches'"
- print >> sys.stderr, " calls 'calls' => create calls
and call_paths table"
- print >> sys.stderr, " callchains 'callchains' => create
call_paths table"
+ printerr("Usage is: export-to-postgresql.py []
[] []")
+ printerr("where:columns 'all' or 'branches'")
+ printerr(" calls 'calls' => create calls and
call_paths table")
+ printerr(" callchains 'callchains' => create
call_paths table")
raise Exception("Too few arguments")
if (len(sys.argv) < 2):
@@ -273,7 +289,7 @@ def do_query(q, s):
return
raise Exception("Query failed: " + q.lastError().text())
-print datetime.datetime.today(), "Creating database..."
+print(datetime.datetime.today(), "Creating database...")
db = QSqlDatabase.addDatabase('QPSQL')
query = QSqlQuery(db)
@@ -506,12 +522,12 @@ do_query(query, 'CREATE VIEW samples_view AS '
' FROM samples')
-file_header = struct.pack("!11sii", "PGCOPY\n\377\r\n\0", 0, 0)
-file_trailer = "\377\377"
+file_header = struct.pack("!11sii", b"PGCOPY\n\377\r\n\0", 0, 0)
+file_trailer = b"\377\377"
def open_output_file(file_name):
path_name = output_dir_name + "/" + file_name
- file = open(path_name, "w+")
+ file = open(path_name, "wb+")
file.write(file_header)
return file
@@ -526,13 +542,13 @@ def copy_output_file_direct(file, table_name):
# Use COPY FROM STDIN because security may prevent postgres from accessing the
files directly
def copy_output_file(file, table_name):
- conn = PQconnectdb("dbname = " + dbname)
+ conn = PQconnectdb(toclientstr("dbname = " + dbname))
if (PQstatus(conn)):
raise Exception("COPY FROM STDIN PQconnectdb failed")
file.write(file_trailer)
file.seek(0)
sql = "COPY " + table_name + " FROM STDIN (FORMAT 'binary')"
- res = PQexec(conn, sql)
+ res = PQexec(conn, toclientstr(sql))
if (PQresultStatus(res) != 4):
raise Exception("COPY FROM STDIN PQexec failed")
data = file.read(65536)
@@ -566,7 +582,7 @@ if perf_db_export_calls:
call_file = open_output_file("call_table.bin")
def trace_begin():
- print datetime.datetime.today(), "Writing to intermediate files..."
+ print(datetime.datetime.today(), "Writing to intermediate files...")
# id == 0 means unknown. It is easier to create records for them than
replace the zeroes with NULLs
evsel_table(0, "unknown")
machine_table(0, 0, "unknown")
@@ -582,7 +598,7 @@ def trace_begin():
unhandled_count = 0
def trace_end():
- print datetime.datetime.today(), "Copying to database..."
+ print(datetime.datetime.today(), "Copy
[PATCH v3 3/4] perf script python: add Python3 support to export-to-sqlite.py
Support both Python2 and Python3 in the export-to-sqlite.py script
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Adrian Hunter
---
tools/perf/scripts/python/export-to-sqlite.py | 23 ++-
1 file changed, 14 insertions(+), 9 deletions(-)
diff --git a/tools/perf/scripts/python/export-to-sqlite.py
b/tools/perf/scripts/python/export-to-sqlite.py
index eb63e6c7107f..3da338243aed 100644
--- a/tools/perf/scripts/python/export-to-sqlite.py
+++ b/tools/perf/scripts/python/export-to-sqlite.py
@@ -10,6 +10,8 @@
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
+from __future__ import print_function
+
import os
import sys
import struct
@@ -60,11 +62,14 @@ perf_db_export_mode = True
perf_db_export_calls = False
perf_db_export_callchains = False
+def printerr(*args, **keyword_args):
+ print(*args, file=sys.stderr, **keyword_args)
+
def usage():
- print >> sys.stderr, "Usage is: export-to-sqlite.py
[] [] []"
- print >> sys.stderr, "where:columns 'all' or 'branches'"
- print >> sys.stderr, " calls 'calls' => create calls
and call_paths table"
- print >> sys.stderr, " callchains 'callchains' => create
call_paths table"
+ printerr("Usage is: export-to-sqlite.py []
[] []");
+ printerr("where:columns 'all' or 'branches'");
+ printerr(" calls 'calls' => create calls and
call_paths table");
+ printerr(" callchains 'callchains' => create
call_paths table");
raise Exception("Too few arguments")
if (len(sys.argv) < 2):
@@ -100,7 +105,7 @@ def do_query_(q):
return
raise Exception("Query failed: " + q.lastError().text())
-print datetime.datetime.today(), "Creating database..."
+print(datetime.datetime.today(), "Creating database ...")
db_exists = False
try:
@@ -378,7 +383,7 @@ if perf_db_export_calls:
call_query.prepare("INSERT INTO calls VALUES (?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?)")
def trace_begin():
- print datetime.datetime.today(), "Writing records..."
+ print(datetime.datetime.today(), "Writing records...")
do_query(query, 'BEGIN TRANSACTION')
# id == 0 means unknown. It is easier to create records for them than
replace the zeroes with NULLs
evsel_table(0, "unknown")
@@ -397,14 +402,14 @@ unhandled_count = 0
def trace_end():
do_query(query, 'END TRANSACTION')
- print datetime.datetime.today(), "Adding indexes"
+ print(datetime.datetime.today(), "Adding indexes")
if perf_db_export_calls:
do_query(query, 'CREATE INDEX pcpid_idx ON calls
(parent_call_path_id)')
do_query(query, 'CREATE INDEX pid_idx ON calls (parent_id)')
if (unhandled_count):
- print datetime.datetime.today(), "Warning: ", unhandled_count,
" unhandled events"
- print datetime.datetime.today(), "Done"
+ print(datetime.datetime.today(), "Warning: ", unhandled_count,
" unhandled events")
+ print(datetime.datetime.today(), "Done")
def trace_unhandled(event_name, context, event_fields_dict):
global unhandled_count
--
2.16.4
[PATCH v3 1/4] perf script python: add Python3 support to exported-sql-viewer.py
Support both Python2 and Python3 in the exported-sql-viewer.py script.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Adrian Hunter
---
tools/perf/scripts/python/exported-sql-viewer.py | 42
1 file changed, 28 insertions(+), 14 deletions(-)
diff --git a/tools/perf/scripts/python/exported-sql-viewer.py
b/tools/perf/scripts/python/exported-sql-viewer.py
index afec9479ca7f..e38518cdcbc3 100755
--- a/tools/perf/scripts/python/exported-sql-viewer.py
+++ b/tools/perf/scripts/python/exported-sql-viewer.py
@@ -88,11 +88,20 @@
#
7fab593ea956 48 89 15 3b 13 22 00movq %rdx,
0x22133b(%rip)
# 8107675243232 2ls 22011 22011 hardware interrupt No
7fab593ea956 _dl_start+0x26 (ld-2.19.so) -> 86a012e0 page_fault
([kernel])
+from __future__ import print_function
+
import sys
import weakref
import threading
import string
-import cPickle
+try:
+ # Python2
+ import cPickle as pickle
+ # size of pickled integer big enough for record size
+ glb_nsz = 8
+except ImportError:
+ import pickle
+ glb_nsz = 16
import re
import os
from PySide.QtCore import *
@@ -102,6 +111,15 @@ from decimal import *
from ctypes import *
from multiprocessing import Process, Array, Value, Event
+# xrange is range in Python3
+try:
+ xrange
+except NameError:
+ xrange = range
+
+def printerr(*args, **keyword_args):
+ print(*args, file=sys.stderr, **keyword_args)
+
# Data formatting helpers
def tohex(ip):
@@ -1004,10 +1022,6 @@ class ChildDataItemFinder():
glb_chunk_sz = 1
-# size of pickled integer big enough for record size
-
-glb_nsz = 8
-
# Background process for SQL data fetcher
class SQLFetcherProcess():
@@ -1066,7 +1080,7 @@ class SQLFetcherProcess():
return True
if space >= glb_nsz:
# Use 0 (or space < glb_nsz) to mean there is
no more at the top of the buffer
- nd = cPickle.dumps(0, cPickle.HIGHEST_PROTOCOL)
+ nd = pickle.dumps(0, pickle.HIGHEST_PROTOCOL)
self.buffer[self.local_head : self.local_head +
len(nd)] = nd
self.local_head = 0
if self.local_tail - self.local_head > sz:
@@ -1084,9 +1098,9 @@ class SQLFetcherProcess():
self.wait_event.wait()
def AddToBuffer(self, obj):
- d = cPickle.dumps(obj, cPickle.HIGHEST_PROTOCOL)
+ d = pickle.dumps(obj, pickle.HIGHEST_PROTOCOL)
n = len(d)
- nd = cPickle.dumps(n, cPickle.HIGHEST_PROTOCOL)
+ nd = pickle.dumps(n, pickle.HIGHEST_PROTOCOL)
sz = n + glb_nsz
self.WaitForSpace(sz)
pos = self.local_head
@@ -1198,12 +1212,12 @@ class SQLFetcher(QObject):
pos = self.local_tail
if len(self.buffer) - pos < glb_nsz:
pos = 0
- n = cPickle.loads(self.buffer[pos : pos + glb_nsz])
+ n = pickle.loads(self.buffer[pos : pos + glb_nsz])
if n == 0:
pos = 0
- n = cPickle.loads(self.buffer[0 : glb_nsz])
+ n = pickle.loads(self.buffer[0 : glb_nsz])
pos += glb_nsz
- obj = cPickle.loads(self.buffer[pos : pos + n])
+ obj = pickle.loads(self.buffer[pos : pos + n])
self.local_tail = pos + n
return obj
@@ -2973,7 +2987,7 @@ class DBRef():
def Main():
if (len(sys.argv) < 2):
- print >> sys.stderr, "Usage is: exported-sql-viewer.py
{ | --help-only}"
+ printerr("Usage is: exported-sql-viewer.py { |
--help-only}");
raise Exception("Too few arguments")
dbname = sys.argv[1]
@@ -2986,8 +3000,8 @@ def Main():
is_sqlite3 = False
try:
- f = open(dbname)
- if f.read(15) == "SQLite format 3":
+ f = open(dbname, "rb")
+ if f.read(15) == b'SQLite format 3':
is_sqlite3 = True
f.close()
except:
--
2.16.4
[PATCH v3 0/4] perf script python: add Python3 support
This is v3 of my version of the patchset. The exporter and viewer changes were all in one patch for V2. For V3 I split them into 3 patches for the reasons below. Patch#1 adds Python3 support to exported-sql-viewer Patch#2 adds Python3 support to export-to-postgresql Patch#3 adds Python3 support to export-to-sqlite Patch#4 cleans up some repeated use of date functions in the code. I added Adrian's signed-off-by to Patch#2 per the revised patch he sent in his review of V2. The content of patches 1, 3 and 4 is unchanged from V2 but I'm including all the patches again to hopefully make your application easier. I added Adrian's Acked-by to Patch#4 per his review of that patch in V2. Once these are applied I believe this is everything needed to run on a system without Python2. Thanks Tony
Re: [PATCH v2 6/7] perf script python: add Python3 support to sql scripts
On 3/6/19 1:26 AM, Adrian Hunter wrote: > On 2/03/19 3:19 AM, Tony Jones wrote: >> Support both Python2 and Python3 in the exported-sql-viewer.py, >> export-to-postgresql.py and export-to-sqlite.py scripts >> >> There may be differences in the ordering of output lines due to >> differences in dictionary ordering etc. However the format within lines >> should be unchanged. >> >> The use of 'from __future__' implies the minimum supported Python2 version >> is now v2.6 >> >> Signed-off-by: Tony Jones >> Signed-off-by: Seeteena Thoufeek >> Cc: Adrian Hunter > > Apart from one issue (see below), it looks good, thank you! > >> --- >> tools/perf/scripts/python/export-to-postgresql.py | 65 >> +++ >> tools/perf/scripts/python/export-to-sqlite.py | 23 >> tools/perf/scripts/python/exported-sql-viewer.py | 42 ++- >> 3 files changed, 84 insertions(+), 46 deletions(-) >> >> diff --git a/tools/perf/scripts/python/export-to-postgresql.py >> b/tools/perf/scripts/python/export-to-postgresql.py >> index 390a351d15ea..439bbbf1e036 100644 >> --- a/tools/perf/scripts/python/export-to-postgresql.py >> +++ b/tools/perf/scripts/python/export-to-postgresql.py >> @@ -10,6 +10,8 @@ >> # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for >> # more details. >> >> +from __future__ import print_function >> + >> import os >> import sys >> import struct >> @@ -199,6 +201,16 @@ import datetime >> >> from PySide.QtSql import * >> >> +if sys.version_info < (3, 0): >> +def tobytes(str): >> +return str >> +else: >> +def tobytes(str): >> +# Use latin-1 (ISO-8859-1) so all code-points 0-255 will result >> +# in one byte (note utf-8 is 2 bytes for values > 128 and >> +# ascii is limited to values <= 128) >> +return bytes(str, "ISO-8859-1") > > Probably this should be the server_encoding, but python2 allowed UTF-8 > so let's just use UTF-8 for now. That will also mean doing the conversion > before getting the len(), otherwise len() can be wrong. I'm not totally understanding what you're saying here. The rationale for using latin-1 and not UTF-8 was clearly expressed in the comment. Else you do indeed run into length issues. Would it be easier, since you have a) more familiarity with the code b) some specific issues I'm not fully understanding if you just took this patch and made the changes you want yourself. I doubt I'll ever use these scripta, my interest is purely in eliminating Python2 as a fixed requirement. Tony
Re: [PATCH v2 5/7] perf script python: add Python3 support to intel-pt-events.py
On 3/5/19 8:10 AM, Tony Jones wrote:
>
> Sure enough, I managed to attach the wrong patch. Sorry. I'll attach revised
> (v3)
> to this thread as the change is trivial.
From: Tony Jones
Date: Tue, 05 Mar 2019 08:31:30 -0800
Subject: [PATCH v3] perf script python: add Python3 support to
intel-pt-events.py
Support both Python2 and Python3 in the intel-pt-events.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Adrian Hunter
---
tools/perf/scripts/python/intel-pt-events.py | 32 ---
1 file changed, 19 insertions(+), 13 deletions(-)
--- a/tools/perf/scripts/python/intel-pt-events.py
+++ b/tools/perf/scripts/python/intel-pt-events.py
@@ -10,6 +10,8 @@
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
+from __future__ import print_function
+
import os
import sys
import struct
@@ -22,34 +24,34 @@ sys.path.append(os.environ['PERF_EXEC_PA
#from Core import *
def trace_begin():
- print "Intel PT Power Events and PTWRITE"
+ print("Intel PT Power Events and PTWRITE")
def trace_end():
- print "End"
+ print("End")
def trace_unhandled(event_name, context, event_fields_dict):
- print ' '.join(['%s=%s'%(k,str(v))for k,v in
sorted(event_fields_dict.items())])
+ print(' '.join(['%s=%s'%(k,str(v))for k,v in
sorted(event_fields_dict.items())]))
def print_ptwrite(raw_buf):
data = struct.unpack_from("> 32) & 0x3
- print "hints: %#x extensions: %#x" % (hints, extensions),
+ print("hints: %#x extensions: %#x" % (hints, extensions), end=' ')
def print_pwre(raw_buf):
data = struct.unpack_from("> 7) & 1
cstate = (payload >> 12) & 0xf
subcstate = (payload >> 8) & 0xf
- print "hw: %u cstate: %u sub-cstate: %u" % (hw, cstate, subcstate),
+ print("hw: %u cstate: %u sub-cstate: %u" % (hw, cstate, subcstate),
+ end=' ')
def print_exstop(raw_buf):
data = struct.unpack_from("> 4) & 0xf
wake_reason = (payload >> 8) & 0xf
- print "deepest cstate: %u last cstate: %u wake reason: %#x" %
(deepest_cstate, last_cstate, wake_reason),
+ print("deepest cstate: %u last cstate: %u wake reason: %#x" %
+ (deepest_cstate, last_cstate, wake_reason), end=' ')
def print_common_start(comm, sample, name):
ts = sample["time"]
cpu = sample["cpu"]
pid = sample["pid"]
tid = sample["tid"]
- print "%16s %5u/%-5u [%03u] %9u.%09u %7s:" % (comm, pid, tid, cpu, ts /
10, ts %10, name),
+ print("%16s %5u/%-5u [%03u] %9u.%09u %7s:" %
+ (comm, pid, tid, cpu, ts / 10, ts %10, name),
+ end=' ')
def print_common_ip(sample, symbol, dso):
ip = sample["ip"]
- print "%16x %s (%s)" % (ip, symbol, dso)
+ print("%16x %s (%s)" % (ip, symbol, dso))
def process_event(param_dict):
event_attr = param_dict["attr"]
@@ -92,12 +98,12 @@ def process_event(param_dict):
name = param_dict["ev_name"]
# Symbol and dso info are not always resolved
- if (param_dict.has_key("dso")):
+ if "dso" in param_dict:
dso = param_dict["dso"]
else:
dso = "[unknown]"
- if (param_dict.has_key("symbol")):
+ if "symbol" in param_dict:
symbol = param_dict["symbol"]
else:
symbol = "[unknown]"
Re: [PATCH v2 5/7] perf script python: add Python3 support to intel-pt-events.py
On 3/5/19 7:02 AM, Tony Jones wrote: > I tested the patch on a Skylake system but it was not connected to our > standard > network so I had to move files manually. It seems I managed to somehow mess > up > and not attach the correct patch :( Sure enough, I managed to attach the wrong patch. Sorry. I'll attach revised (v3) to this thread as the change is trivial. I am seeing a lot of "instruction trace errors" on this Skylake system. I see them using Python2 without any of the changes. I've not looked at this. Output of v3. # head /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 94 model name : Intel(R) Xeon(R) CPU E3-1240 v5 @ 3.50GHz stepping: 3 microcode : 0xc6 cpu MHz : 1600.887 cache size : 8192 KB physical id : 0 # ldd /tmp/perf/bin/perf | grep python libpython3.6m.so.1.0 => /usr/lib64/libpython3.6m.so.1.0 (0x7fd8fdaae000) # /tmp/perf/bin/perf script report intel-pt-events -i /tmp/perf.data | tail -20 swapper 0/0 [006] 17424.535324338 cbr: 27 freq: 2703 MHz ( 77%)0 [unknown] ([unknown]) perf 20899/20899 [003] 17424.535324363 cbr: 27 freq: 2703 MHz ( 77%)0 [unknown] ([unknown]) swapper 0/0 [004] 17424.535324387 cbr: 27 freq: 2703 MHz ( 77%)0 [unknown] ([unknown]) swapper 0/0 [000] 17424.535324867 cbr: 27 freq: 2703 MHz ( 77%)0 [unknown] ([unknown]) swapper 0/0 [006] 17424.543315376 cbr: 24 freq: 2403 MHz ( 69%)0 [unknown] ([unknown]) perf 20899/20899 [003] 17424.699315686 cbr: 16 freq: 1602 MHz ( 46%)0 [unknown] ([unknown]) swapper 0/0 [006] 17424.714915068 cbr: 16 freq: 1602 MHz ( 46%)0 [unknown] ([unknown]) swapper 0/0 [005] 17424.779321732 cbr: 9 freq: 901 MHz ( 26%)0 [unknown] ([unknown]) swapper 0/0 [001] 17424.779322738 cbr: 9 freq: 901 MHz ( 26%)0 [unknown] ([unknown]) swapper 0/0 [003] 17424.903321958 cbr: 9 freq: 901 MHz ( 26%)0 [unknown] ([unknown]) swapper 0/0 [000] 17425.099316258 cbr: 16 freq: 1602 MHz ( 46%)0 [unknown] ([unknown]) swapper 0/0 [003] 17425.107316167 cbr: 16 freq: 1602 MHz ( 46%)0 [unknown] ([unknown]) swapper 0/0 [004] 17425.519574381 cbr: 16 freq: 1602 MHz ( 46%)0 [unknown] ([unknown]) swapper 0/0 [001] 17425.519796998 cbr: 16 freq: 1602 MHz ( 46%)0 [unknown] ([unknown]) swapper 0/0 [002] 17425.519828095 cbr: 16 freq: 1602 MHz ( 46%)0 [unknown] ([unknown]) swapper 0/0 [005] 17425.519897417 cbr: 16 freq: 1602 MHz ( 46%)0 [unknown] ([unknown]) swapper 0/0 [007] 17425.519936473 cbr: 16 freq: 1602 MHz ( 46%)0 [unknown] ([unknown]) Warning: 260 instruction trace errors End
Re: [PATCH v2 0/7] perf script python: add Python3 support
On 3/5/19 6:53 AM, Tony Jones wrote:
> On 3/5/19 1:55 AM, Adrian Hunter wrote:
>
>> perf tools link against python2 so have the scripts been tested with python3?
>
> Not if PYTHON=python3 is specified on the build line. Yes, of course they
> were tested.
See below for exporter. The viewer also works but obviously harder to
demonstrate via log.
$ make PYTHON=python3 prefix=/tmp/perf install
$ ldd /tmp/perf/bin/perf | grep python
libpython3.6m.so.1.0 => /usr/lib64/libpython3.6m.so.1.0
(0x7f90d32c3000)
# /tmp/perf/bin/perf record -o /tmp/perf.data -e cycles,instructions,branches
/bin/false
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.017 MB /tmp/perf.data (39 samples) ]
postgres@svr1:~> id
uid=26(postgres) gid=26(postgres) groups=26(postgres)
postgres@svr1:~> ldd /tmp/perf/bin/perf | grep python
libpython3.6m.so.1.0 => /usr/lib64/libpython3.6m.so.1.0
(0x7f7714156000)
postgres@svr1:~> /tmp/perf/bin/perf script -i /tmp/perf.data -s
/tmp/perf/libexec/perf-core/scripts/python/export-to-postgresql.py db1
2019-03-05 07:14:04.243419 Creating database...
This version of PostgreSQL is not supported and may not work.
This version of PostgreSQL is not supported and may not work.
2019-03-05 07:14:05.122981 Writing to intermediate files...
2019-03-05 07:14:05.187927 Copying to database...
2019-03-05 07:14:05.72 Removing intermediate files...
2019-03-05 07:14:05.222766 Adding primary keys
2019-03-05 07:14:05.714776 Adding foreign keys
postgres@svr1:~> pg_dump -a -t public.comms -t public.dsos db1
--
-- PostgreSQL database dump
--
-- Dumped from database version 10.6
-- Dumped by pg_dump version 10.6
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET client_min_messages = warning;
SET row_security = off;
--
-- Data for Name: comms; Type: TABLE DATA; Schema: public; Owner: postgres
--
COPY public.comms (id, comm) FROM stdin;
0 unknown
1 perf
2 false
\.
--
-- Data for Name: dsos; Type: TABLE DATA; Schema: public; Owner: postgres
--
COPY public.dsos (id, machine_id, short_name, long_name, build_id) FROM stdin;
0 0 unknown unknown
1 1 [kernel.kallsyms] [kernel.kallsyms]
0621827fb34281c413e26a0c2d6ba1c0abd706b9
2 1 ld-2.26.so /lib64/ld-2.26.so
bca321ae4ce1ab788b186e2085f60c1c9723cc14
\.
--
-- PostgreSQL database dump complete
--
Re: [PATCH v2 5/7] perf script python: add Python3 support to intel-pt-events.py
On 3/5/19 2:16 AM, Adrian Hunter wrote: > On 2/03/19 3:19 AM, Tony Jones wrote: >> Support both Python2 and Python3 in the intel-pt-events.py script >> >> There may be differences in the ordering of output lines due to >> differences in dictionary ordering etc. However the format within lines >> should be unchanged. >> >> The use of 'from __future__' implies the minimum supported Python2 version >> is now v2.6 >> >> Signed-off-by: Tony Jones >> Signed-off-by: Seeteena Thoufeek >> Cc: Adrian Hunter > > One change missed, see below, otherwise: I tested the patch on a Skylake system but it was not connected to our standard network so I had to move files manually. It seems I managed to somehow mess up and not attach the correct patch :( >> def trace_unhandled(event_name, context, event_fields_dict): >> print ' '.join(['%s=%s'%(k,str(v))for k,v in >> sorted(event_fields_dict.items())]) > > Also above line You are correct. Apologies. I'll send a revised version. tony
Re: [PATCH v2 0/7] perf script python: add Python3 support
On 3/5/19 1:55 AM, Adrian Hunter wrote: > perf tools link against python2 so have the scripts been tested with python3? Not if PYTHON=python3 is specified on the build line. Yes, of course they were tested.
[PATCH v2 6/7] perf script python: add Python3 support to sql scripts
Support both Python2 and Python3 in the exported-sql-viewer.py,
export-to-postgresql.py and export-to-sqlite.py scripts
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Adrian Hunter
---
tools/perf/scripts/python/export-to-postgresql.py | 65 +++
tools/perf/scripts/python/export-to-sqlite.py | 23
tools/perf/scripts/python/exported-sql-viewer.py | 42 ++-
3 files changed, 84 insertions(+), 46 deletions(-)
diff --git a/tools/perf/scripts/python/export-to-postgresql.py
b/tools/perf/scripts/python/export-to-postgresql.py
index 390a351d15ea..439bbbf1e036 100644
--- a/tools/perf/scripts/python/export-to-postgresql.py
+++ b/tools/perf/scripts/python/export-to-postgresql.py
@@ -10,6 +10,8 @@
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
+from __future__ import print_function
+
import os
import sys
import struct
@@ -199,6 +201,16 @@ import datetime
from PySide.QtSql import *
+if sys.version_info < (3, 0):
+ def tobytes(str):
+ return str
+else:
+ def tobytes(str):
+ # Use latin-1 (ISO-8859-1) so all code-points 0-255 will result
+ # in one byte (note utf-8 is 2 bytes for values > 128 and
+ # ascii is limited to values <= 128)
+ return bytes(str, "ISO-8859-1")
+
# Need to access PostgreSQL C library directly to use COPY FROM STDIN
from ctypes import *
libpq = CDLL("libpq.so.5")
@@ -234,12 +246,14 @@ perf_db_export_mode = True
perf_db_export_calls = False
perf_db_export_callchains = False
+def printerr(*args, **kw_args):
+ print(*args, file=sys.stderr, **kw_args)
def usage():
- print >> sys.stderr, "Usage is: export-to-postgresql.py
[] [] []"
- print >> sys.stderr, "where:columns 'all' or 'branches'"
- print >> sys.stderr, " calls 'calls' => create calls
and call_paths table"
- print >> sys.stderr, " callchains 'callchains' => create
call_paths table"
+ printerr("Usage is: export-to-postgresql.py []
[] []")
+ printerr("where:columns 'all' or 'branches'")
+ printerr(" calls 'calls' => create calls and
call_paths table")
+ printerr(" callchains 'callchains' => create
call_paths table")
raise Exception("Too few arguments")
if (len(sys.argv) < 2):
@@ -273,7 +287,7 @@ def do_query(q, s):
return
raise Exception("Query failed: " + q.lastError().text())
-print datetime.datetime.today(), "Creating database..."
+print(datetime.datetime.today(), "Creating database...")
db = QSqlDatabase.addDatabase('QPSQL')
query = QSqlQuery(db)
@@ -506,12 +520,12 @@ do_query(query, 'CREATE VIEW samples_view AS '
' FROM samples')
-file_header = struct.pack("!11sii", "PGCOPY\n\377\r\n\0", 0, 0)
-file_trailer = "\377\377"
+file_header = struct.pack("!11sii", tobytes("PGCOPY\n\377\r\n\0"), 0, 0)
+file_trailer = tobytes("\377\377")
def open_output_file(file_name):
path_name = output_dir_name + "/" + file_name
- file = open(path_name, "w+")
+ file = open(path_name, "wb+")
file.write(file_header)
return file
@@ -526,13 +540,13 @@ def copy_output_file_direct(file, table_name):
# Use COPY FROM STDIN because security may prevent postgres from accessing the
files directly
def copy_output_file(file, table_name):
- conn = PQconnectdb("dbname = " + dbname)
+ conn = PQconnectdb(tobytes("dbname = " + dbname))
if (PQstatus(conn)):
raise Exception("COPY FROM STDIN PQconnectdb failed")
file.write(file_trailer)
file.seek(0)
sql = "COPY " + table_name + " FROM STDIN (FORMAT 'binary')"
- res = PQexec(conn, sql)
+ res = PQexec(conn, tobytes(sql))
if (PQresultStatus(res) != 4):
raise Exception("COPY FROM STDIN PQexec failed")
data = file.read(65536)
@@ -566,7 +580,7 @@ if perf_db_export_calls:
call_file = open_output_file("call_table.bin")
def trace_begin():
- print datetime.datetime.today(), "Writing to intermediate files..."
+ print(datetime.datetime.today(), "Writing to intermediate files...")
# id == 0 means unknown. It is easier to cr
[PATCH v2 5/7] perf script python: add Python3 support to intel-pt-events.py
Support both Python2 and Python3 in the intel-pt-events.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Adrian Hunter
---
tools/perf/scripts/python/intel-pt-events.py | 30 +---
1 file changed, 18 insertions(+), 12 deletions(-)
diff --git a/tools/perf/scripts/python/intel-pt-events.py
b/tools/perf/scripts/python/intel-pt-events.py
index 2177722f509e..5c42c4c359dc 100644
--- a/tools/perf/scripts/python/intel-pt-events.py
+++ b/tools/perf/scripts/python/intel-pt-events.py
@@ -10,6 +10,8 @@
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
+from __future__ import print_function
+
import os
import sys
import struct
@@ -22,10 +24,10 @@ sys.path.append(os.environ['PERF_EXEC_PATH'] + \
#from Core import *
def trace_begin():
- print "Intel PT Power Events and PTWRITE"
+ print("Intel PT Power Events and PTWRITE")
def trace_end():
- print "End"
+ print("End")
def trace_unhandled(event_name, context, event_fields_dict):
print ' '.join(['%s=%s'%(k,str(v))for k,v in
sorted(event_fields_dict.items())])
@@ -35,21 +37,21 @@ def print_ptwrite(raw_buf):
flags = data[0]
payload = data[1]
exact_ip = flags & 1
- print "IP: %u payload: %#x" % (exact_ip, payload),
+ print("IP: %u payload: %#x" % (exact_ip, payload), end=' ')
def print_cbr(raw_buf):
data = struct.unpack_from("> 32) & 0x3
- print "hints: %#x extensions: %#x" % (hints, extensions),
+ print("hints: %#x extensions: %#x" % (hints, extensions), end=' ')
def print_pwre(raw_buf):
data = struct.unpack_from("> 7) & 1
cstate = (payload >> 12) & 0xf
subcstate = (payload >> 8) & 0xf
- print "hw: %u cstate: %u sub-cstate: %u" % (hw, cstate, subcstate),
+ print("hw: %u cstate: %u sub-cstate: %u" % (hw, cstate, subcstate),
+ end=' ')
def print_exstop(raw_buf):
data = struct.unpack_from("> 4) & 0xf
wake_reason = (payload >> 8) & 0xf
- print "deepest cstate: %u last cstate: %u wake reason: %#x" %
(deepest_cstate, last_cstate, wake_reason),
+ print("deepest cstate: %u last cstate: %u wake reason: %#x" %
+ (deepest_cstate, last_cstate, wake_reason), end=' ')
def print_common_start(comm, sample, name):
ts = sample["time"]
cpu = sample["cpu"]
pid = sample["pid"]
tid = sample["tid"]
- print "%16s %5u/%-5u [%03u] %9u.%09u %7s:" % (comm, pid, tid, cpu, ts /
10, ts %10, name),
+ print("%16s %5u/%-5u [%03u] %9u.%09u %7s:" %
+ (comm, pid, tid, cpu, ts / 10, ts %10, name),
+ end=' ')
def print_common_ip(sample, symbol, dso):
ip = sample["ip"]
- print "%16x %s (%s)" % (ip, symbol, dso)
+ print("%16x %s (%s)" % (ip, symbol, dso))
def process_event(param_dict):
event_attr = param_dict["attr"]
@@ -92,12 +98,12 @@ def process_event(param_dict):
name = param_dict["ev_name"]
# Symbol and dso info are not always resolved
- if (param_dict.has_key("dso")):
+ if "dso" in param_dict:
dso = param_dict["dso"]
else:
dso = "[unknown]"
- if (param_dict.has_key("symbol")):
+ if "symbol" in param_dict:
symbol = param_dict["symbol"]
else:
symbol = "[unknown]"
--
2.16.4
[PATCH v2 4/7] perf script python: add Python3 support to event_analyzing_sample.py
Support both Python2 and Python3 in the event_analyzing_sample.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Feng Tang
---
.../perf/scripts/python/event_analyzing_sample.py | 48 +++---
1 file changed, 25 insertions(+), 23 deletions(-)
diff --git a/tools/perf/scripts/python/event_analyzing_sample.py
b/tools/perf/scripts/python/event_analyzing_sample.py
index 2ec8915b74c5..aa1e2cfa26a6 100644
--- a/tools/perf/scripts/python/event_analyzing_sample.py
+++ b/tools/perf/scripts/python/event_analyzing_sample.py
@@ -15,6 +15,8 @@
# for a x86 HW PMU event: PEBS with load latency data.
#
+from __future__ import print_function
+
import os
import sys
import math
@@ -37,7 +39,7 @@ con = sqlite3.connect("/dev/shm/perf.db")
con.isolation_level = None
def trace_begin():
-print "In trace_begin:\n"
+print("In trace_begin:\n")
#
# Will create several tables at the start, pebs_ll is for PEBS data
with
@@ -76,12 +78,12 @@ def process_event(param_dict):
name = param_dict["ev_name"]
# Symbol and dso info are not always resolved
-if (param_dict.has_key("dso")):
+if ("dso" in param_dict):
dso = param_dict["dso"]
else:
dso = "Unknown_dso"
-if (param_dict.has_key("symbol")):
+if ("symbol" in param_dict):
symbol = param_dict["symbol"]
else:
symbol = "Unknown_symbol"
@@ -102,7 +104,7 @@ def insert_db(event):
event.ip, event.status, event.dse, event.dla,
event.lat))
def trace_end():
-print "In trace_end:\n"
+print("In trace_end:\n")
# We show the basic info for the 2 type of event classes
show_general_events()
show_pebs_ll()
@@ -123,29 +125,29 @@ def show_general_events():
# Check the total record number in the table
count = con.execute("select count(*) from gen_events")
for t in count:
-print "There is %d records in gen_events table" % t[0]
+print("There is %d records in gen_events table" % t[0])
if t[0] == 0:
return
-print "Statistics about the general events grouped by
thread/symbol/dso: \n"
+print("Statistics about the general events grouped by
thread/symbol/dso: \n")
# Group by thread
commq = con.execute("select comm, count(comm) from gen_events group by
comm order by -count(comm)")
-print "\n%16s %8s %16s\n%s" % ("comm", "number", "histogram", "="*42)
+print("\n%16s %8s %16s\n%s" % ("comm", "number", "histogram", "="*42))
for row in commq:
- print "%16s %8d %s" % (row[0], row[1], num2sym(row[1]))
+ print("%16s %8d %s" % (row[0], row[1], num2sym(row[1])))
# Group by symbol
-print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histogram", "="*58)
+print("\n%32s %8s %16s\n%s" % ("symbol", "number", "histogram",
"="*58))
symbolq = con.execute("select symbol, count(symbol) from gen_events
group by symbol order by -count(symbol)")
for row in symbolq:
- print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))
+ print("%32s %8d %s" % (row[0], row[1], num2sym(row[1])))
# Group by dso
-print "\n%40s %8s %16s\n%s" % ("dso", "number", "histogram", "="*74)
+print("\n%40s %8s %16s\n%s" % ("dso", "number", "histogram", "="*74))
dsoq = con.execute("select dso, count(dso) from gen_events group by
dso order by -count(dso)")
for row in dsoq:
- print "%40s %8d %s" % (row[0], row[1], num2sym(row[1]))
+ print("%40s %8d %s" % (row[0], row[1], num2sym(row[1])))
#
# This function just shows the basic info, and we could do more with the
@@ -156,35 +158,35 @@ def show_pebs_ll():
count = con.execute("select count(*) from pebs_ll")
for t in count:
-print "There is %d records in pebs_ll table" % t[0]
+
[PATCH v2 7/7] perf script python: add printdate function to SQL exporters
Introduce a printdate function to eliminate the repetitive use of
datetime.datetime.today() in the SQL exporting scripts.
Signed-off-by: Tony Jones
Cc: Adrian Hunter
---
tools/perf/scripts/python/export-to-postgresql.py | 19 +++
tools/perf/scripts/python/export-to-sqlite.py | 13 -
2 files changed, 19 insertions(+), 13 deletions(-)
diff --git a/tools/perf/scripts/python/export-to-postgresql.py
b/tools/perf/scripts/python/export-to-postgresql.py
index 439bbbf1e036..515dc5506427 100644
--- a/tools/perf/scripts/python/export-to-postgresql.py
+++ b/tools/perf/scripts/python/export-to-postgresql.py
@@ -249,6 +249,9 @@ perf_db_export_callchains = False
def printerr(*args, **kw_args):
print(*args, file=sys.stderr, **kw_args)
+def printdate(*args, **kw_args):
+print(datetime.datetime.today(), *args, sep=' ', **kw_args)
+
def usage():
printerr("Usage is: export-to-postgresql.py []
[] []")
printerr("where:columns 'all' or 'branches'")
@@ -287,7 +290,7 @@ def do_query(q, s):
return
raise Exception("Query failed: " + q.lastError().text())
-print(datetime.datetime.today(), "Creating database...")
+printdate("Creating database...")
db = QSqlDatabase.addDatabase('QPSQL')
query = QSqlQuery(db)
@@ -580,7 +583,7 @@ if perf_db_export_calls:
call_file = open_output_file("call_table.bin")
def trace_begin():
- print(datetime.datetime.today(), "Writing to intermediate files...")
+ printdate("Writing to intermediate files...")
# id == 0 means unknown. It is easier to create records for them than
replace the zeroes with NULLs
evsel_table(0, "unknown")
machine_table(0, 0, "unknown")
@@ -596,7 +599,7 @@ def trace_begin():
unhandled_count = 0
def trace_end():
- print(datetime.datetime.today(), "Copying to database...")
+ printdate("Copying to database...")
copy_output_file(evsel_file,"selected_events")
copy_output_file(machine_file, "machines")
copy_output_file(thread_file, "threads")
@@ -611,7 +614,7 @@ def trace_end():
if perf_db_export_calls:
copy_output_file(call_file, "calls")
- print(datetime.datetime.today(), "Removing intermediate files...")
+ printdate("Removing intermediate files...")
remove_output_file(evsel_file)
remove_output_file(machine_file)
remove_output_file(thread_file)
@@ -626,7 +629,7 @@ def trace_end():
if perf_db_export_calls:
remove_output_file(call_file)
os.rmdir(output_dir_name)
- print(datetime.datetime.today(), "Adding primary keys")
+ printdate("Adding primary keys")
do_query(query, 'ALTER TABLE selected_events ADD PRIMARY KEY (id)')
do_query(query, 'ALTER TABLE machinesADD PRIMARY KEY (id)')
do_query(query, 'ALTER TABLE threads ADD PRIMARY KEY (id)')
@@ -641,7 +644,7 @@ def trace_end():
if perf_db_export_calls:
do_query(query, 'ALTER TABLE calls ADD PRIMARY KEY
(id)')
- print(datetime.datetime.today(), "Adding foreign keys")
+ printdate("Adding foreign keys")
do_query(query, 'ALTER TABLE threads '
'ADD CONSTRAINT machinefk FOREIGN KEY
(machine_id) REFERENCES machines (id),'
'ADD CONSTRAINT processfk FOREIGN KEY
(process_id) REFERENCES threads(id)')
@@ -677,8 +680,8 @@ def trace_end():
do_query(query, 'CREATE INDEX pid_idx ON calls (parent_id)')
if (unhandled_count):
- print(datetime.datetime.today(), "Warning: ", unhandled_count,
" unhandled events")
- print(datetime.datetime.today(), "Done")
+ printdate("Warning: ", unhandled_count, " unhandled events")
+ printdate("Done")
def trace_unhandled(event_name, context, event_fields_dict):
global unhandled_count
diff --git a/tools/perf/scripts/python/export-to-sqlite.py
b/tools/perf/scripts/python/export-to-sqlite.py
index 3da338243aed..3b71902a5a21 100644
--- a/tools/perf/scripts/python/export-to-sqlite.py
+++ b/tools/perf/scripts/python/export-to-sqlite.py
@@ -65,6 +65,9 @@ perf_db_export_callchains = False
def printerr(*args, **keyword_args):
print(*args, file=sys.stderr, **keyword_args)
+def printdate(*args, **kw_args):
+print(datetime.datetime.today(), *args, sep=' ', **kw_args)
+
def usage():
printerr("Usage is: export-to-sqlite.py []
[] []");
printerr("where:columns
[PATCH v2 3/7] perf script python: add Python3 support to check-perf-trace.py
Support both Python 2 and Python 3 in the check-perf-trace.py script.
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of from __future__ implies the minimum supported version of
Python2 is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Tom Zanussi
---
tools/perf/scripts/python/check-perf-trace.py | 31 +++
1 file changed, 17 insertions(+), 14 deletions(-)
diff --git a/tools/perf/scripts/python/check-perf-trace.py
b/tools/perf/scripts/python/check-perf-trace.py
index f4838db3e518..d2c22954800d 100644
--- a/tools/perf/scripts/python/check-perf-trace.py
+++ b/tools/perf/scripts/python/check-perf-trace.py
@@ -7,6 +7,8 @@
# events, etc. Basically, if this script runs successfully and
# displays expected results, Python scripting support should be ok.
+from __future__ import print_function
+
import os
import sys
@@ -19,7 +21,7 @@ from perf_trace_context import *
unhandled = autodict()
def trace_begin():
- print "trace_begin"
+ print("trace_begin")
pass
def trace_end():
@@ -33,7 +35,7 @@ def irq__softirq_entry(event_name, context, common_cpu,
print_uncommon(context)
- print "vec=%s\n" % (symbol_str("irq__softirq_entry", "vec", vec)),
+ print("vec=%s" % (symbol_str("irq__softirq_entry", "vec", vec)))
def kmem__kmalloc(event_name, context, common_cpu,
common_secs, common_nsecs, common_pid, common_comm,
@@ -44,10 +46,10 @@ def kmem__kmalloc(event_name, context, common_cpu,
print_uncommon(context)
- print "call_site=%u, ptr=%u, bytes_req=%u, " \
- "bytes_alloc=%u, gfp_flags=%s\n" % \
+ print("call_site=%u, ptr=%u, bytes_req=%u, "
+ "bytes_alloc=%u, gfp_flags=%s" %
(call_site, ptr, bytes_req, bytes_alloc,
- flag_str("kmem__kmalloc", "gfp_flags", gfp_flags)),
+ flag_str("kmem__kmalloc", "gfp_flags", gfp_flags)))
def trace_unhandled(event_name, context, event_fields_dict):
try:
@@ -56,26 +58,27 @@ def trace_unhandled(event_name, context, event_fields_dict):
unhandled[event_name] = 1
def print_header(event_name, cpu, secs, nsecs, pid, comm):
- print "%-20s %5u %05u.%09u %8u %-20s " % \
+ print("%-20s %5u %05u.%09u %8u %-20s " %
(event_name, cpu, secs, nsecs, pid, comm),
+ end=' ')
# print trace fields not included in handler args
def print_uncommon(context):
- print "common_preempt_count=%d, common_flags=%s, " \
- "common_lock_depth=%d, " % \
+ print("common_preempt_count=%d, common_flags=%s, "
+ "common_lock_depth=%d, " %
(common_pc(context), trace_flag_str(common_flags(context)),
- common_lock_depth(context))
+ common_lock_depth(context)))
def print_unhandled():
keys = unhandled.keys()
if not keys:
return
- print "\nunhandled events:\n\n",
+ print("\nunhandled events:\n")
- print "%-40s %10s\n" % ("event", "count"),
- print "%-40s %10s\n" % ("", \
- "---"),
+ print("%-40s %10s" % ("event", "count"))
+ print("%-40s %10s" % ("",
+ "---"))
for event_name in keys:
- print "%-40s %10d\n" % (event_name, unhandled[event_name])
+ print("%-40s %10d\n" % (event_name, unhandled[event_name]))
--
2.16.4
[PATCH v2 0/7] perf script python: add Python3 support
This is v2 of my version of the patchset. Incorporating the previous feedback. Some changes from v1 were already merged. Patch 1/7 deals with the existing inconsistent indentation. Indentation is now consistent per file but varying styles (tabs, 4 spaces and 8 spaces). I will followup at a later date with changes to checkpatch to ensure that the syntax per file is maintained. Patches 2/7 through 5/7 were sent in v1, they have been changed to remove the previous indentation changes Patch 6/7 was sent in v1. I had previously *not* been able to test export-to-postgresql.py. I was able to do so this time and found that more changes were needed.The author of the original code seems concerned about code-style so I would suggest you only merge with his explicit ACK. Patch 7/7 was not in v1, it cleans up some repeated use of date functions in the SQL exporters. It is not mandatory for Python3 support. It is dependent on Patch#6. I hope I've got everything correct, I've retested until I feel I can't look at Python code anymore for a while :-). Hopefully I've not made any more mistakes. If I have, please LMK and I'll do v3. Thanks Tony
[PATCH v2 2/7] perf script python: add Python3 support to futex-contention.py
Support both Python2 and Python3 in the futex-contention.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/futex-contention.py | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/tools/perf/scripts/python/futex-contention.py
b/tools/perf/scripts/python/futex-contention.py
index f221c62e0a10..0c4841acf75d 100644
--- a/tools/perf/scripts/python/futex-contention.py
+++ b/tools/perf/scripts/python/futex-contention.py
@@ -10,6 +10,8 @@
#
# Measures futex contention
+from __future__ import print_function
+
import os, sys
sys.path.append(os.environ['PERF_EXEC_PATH'] +
'/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
from Util import *
@@ -33,18 +35,18 @@ def syscalls__sys_enter_futex(event, ctxt, cpu, s, ns, tid,
comm, callchain,
def syscalls__sys_exit_futex(event, ctxt, cpu, s, ns, tid, comm, callchain,
nr, ret):
- if thread_blocktime.has_key(tid):
+ if tid in thread_blocktime:
elapsed = nsecs(s, ns) - thread_blocktime[tid]
add_stats(lock_waits, (tid, thread_thislock[tid]), elapsed)
del thread_blocktime[tid]
del thread_thislock[tid]
def trace_begin():
- print "Press control+C to stop and show the summary"
+ print("Press control+C to stop and show the summary")
def trace_end():
for (tid, lock) in lock_waits:
min, max, avg, count = lock_waits[tid, lock]
- print "%s[%d] lock %x contended %d times, %d avg ns" % \
- (process_names[tid], tid, lock, count, avg)
+ print("%s[%d] lock %x contended %d times, %d avg ns" %
+ (process_names[tid], tid, lock, count, avg))
--
2.16.4
[PATCH v2 1/7] perf script python: remove mixed indentation
Remove mixed indentation in Python scripts. Revert to either all
tabs (most common form) or all spaces (4 or 8) depending on what
was the intent of the original commit. This is necessary to
complete Python3 support as it will flag an error if it encounters
mixed indentation.
Signed-off-by: Tony Jones
---
tools/perf/scripts/python/check-perf-trace.py | 65 +++---
tools/perf/scripts/python/compaction-times.py | 8 +--
.../perf/scripts/python/event_analyzing_sample.py | 6 +-
.../perf/scripts/python/failed-syscalls-by-pid.py | 38 ++---
tools/perf/scripts/python/futex-contention.py | 2 +-
tools/perf/scripts/python/intel-pt-events.py | 32 +--
tools/perf/scripts/python/mem-phys-addr.py | 7 ++-
tools/perf/scripts/python/net_dropmonitor.py | 2 +-
tools/perf/scripts/python/netdev-times.py | 12 ++--
tools/perf/scripts/python/sched-migration.py | 6 +-
tools/perf/scripts/python/sctop.py | 13 +++--
tools/perf/scripts/python/stackcollapse.py | 2 +-
tools/perf/scripts/python/syscall-counts-by-pid.py | 47
tools/perf/scripts/python/syscall-counts.py| 31 +--
14 files changed, 136 insertions(+), 135 deletions(-)
diff --git a/tools/perf/scripts/python/check-perf-trace.py
b/tools/perf/scripts/python/check-perf-trace.py
index 334599c6032c..f4838db3e518 100644
--- a/tools/perf/scripts/python/check-perf-trace.py
+++ b/tools/perf/scripts/python/check-perf-trace.py
@@ -23,60 +23,59 @@ def trace_begin():
pass
def trace_end():
-print_unhandled()
+ print_unhandled()
def irq__softirq_entry(event_name, context, common_cpu,
- common_secs, common_nsecs, common_pid, common_comm,
- common_callchain, vec):
- print_header(event_name, common_cpu, common_secs, common_nsecs,
- common_pid, common_comm)
+ common_secs, common_nsecs, common_pid, common_comm,
+ common_callchain, vec):
+ print_header(event_name, common_cpu, common_secs, common_nsecs,
+ common_pid, common_comm)
-print_uncommon(context)
+ print_uncommon(context)
- print "vec=%s\n" % \
- (symbol_str("irq__softirq_entry", "vec", vec)),
+ print "vec=%s\n" % (symbol_str("irq__softirq_entry", "vec", vec)),
def kmem__kmalloc(event_name, context, common_cpu,
- common_secs, common_nsecs, common_pid, common_comm,
- common_callchain, call_site, ptr, bytes_req, bytes_alloc,
- gfp_flags):
- print_header(event_name, common_cpu, common_secs, common_nsecs,
- common_pid, common_comm)
+ common_secs, common_nsecs, common_pid, common_comm,
+ common_callchain, call_site, ptr, bytes_req, bytes_alloc,
+ gfp_flags):
+ print_header(event_name, common_cpu, common_secs, common_nsecs,
+ common_pid, common_comm)
-print_uncommon(context)
+ print_uncommon(context)
- print "call_site=%u, ptr=%u, bytes_req=%u, " \
+ print "call_site=%u, ptr=%u, bytes_req=%u, " \
"bytes_alloc=%u, gfp_flags=%s\n" % \
(call_site, ptr, bytes_req, bytes_alloc,
-
flag_str("kmem__kmalloc", "gfp_flags", gfp_flags)),
def trace_unhandled(event_name, context, event_fields_dict):
-try:
-unhandled[event_name] += 1
-except TypeError:
-unhandled[event_name] = 1
+ try:
+ unhandled[event_name] += 1
+ except TypeError:
+ unhandled[event_name] = 1
def print_header(event_name, cpu, secs, nsecs, pid, comm):
print "%-20s %5u %05u.%09u %8u %-20s " % \
- (event_name, cpu, secs, nsecs, pid, comm),
+ (event_name, cpu, secs, nsecs, pid, comm),
# print trace fields not included in handler args
def print_uncommon(context):
-print "common_preempt_count=%d, common_flags=%s, common_lock_depth=%d, " \
-% (common_pc(context), trace_flag_str(common_flags(context)), \
- common_lock_depth(context))
+ print "common_preempt_count=%d, common_flags=%s, " \
+ "common_lock_depth=%d, " % \
+ (common_pc(context), trace_flag_str(common_flags(context)),
+ common_lock_depth(context))
def print_unhandled():
-keys = unhandled.keys()
-if not keys:
-return
+ keys = unhandled.keys()
+ if not keys:
+ return
-print "\nunhandled events:\n\n",
+ print "\nunhandled events:\n\n",
-print "%-40s %10s\n" % ("event", "count"),
-p
[tip:perf/core] perf script python: Add Python3 support to syscall-counts-by-pid.py
Commit-ID: de667cce7f4f96b6e22da8fd9c065b961f355080
Gitweb: https://git.kernel.org/tip/de667cce7f4f96b6e22da8fd9c065b961f355080
Author: Tony Jones
AuthorDate: Fri, 22 Feb 2019 15:06:18 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Mon, 25 Feb 2019 17:17:13 -0300
perf script python: Add Python3 support to syscall-counts-by-pid.py
Support both Python2 and Python3 in the syscall-counts-by-pid.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Link: http://lkml.kernel.org/r/2019030619.17887-15-to...@suse.de
Signed-off-by: Seeteena Thoufeek
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/syscall-counts-by-pid.py | 22 --
1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/tools/perf/scripts/python/syscall-counts-by-pid.py
b/tools/perf/scripts/python/syscall-counts-by-pid.py
index daf314cc5dd3..42782487b0e9 100644
--- a/tools/perf/scripts/python/syscall-counts-by-pid.py
+++ b/tools/perf/scripts/python/syscall-counts-by-pid.py
@@ -5,6 +5,8 @@
# Displays system-wide system call totals, broken down by syscall.
# If a [comm] arg is specified, only syscalls called by [comm] are displayed.
+from __future__ import print_function
+
import os, sys
sys.path.append(os.environ['PERF_EXEC_PATH'] + \
@@ -31,7 +33,7 @@ if len(sys.argv) > 1:
syscalls = autodict()
def trace_begin():
- print "Press control+C to stop and show the summary"
+ print("Press control+C to stop and show the summary")
def trace_end():
print_syscall_totals()
@@ -55,20 +57,20 @@ def syscalls__sys_enter(event_name, context, common_cpu,
def print_syscall_totals():
if for_comm is not None:
- print "\nsyscall events for %s:\n\n" % (for_comm),
+ print("\nsyscall events for %s:\n" % (for_comm))
else:
- print "\nsyscall events by comm/pid:\n\n",
+ print("\nsyscall events by comm/pid:\n")
-print "%-40s %10s\n" % ("comm [pid]/syscalls", "count"),
-print "%-40s %10s\n" % ("", \
- "--"),
+print("%-40s %10s" % ("comm [pid]/syscalls", "count"))
+print("%-40s %10s" % ("",
+"--"))
comm_keys = syscalls.keys()
for comm in comm_keys:
pid_keys = syscalls[comm].keys()
for pid in pid_keys:
- print "\n%s [%d]\n" % (comm, pid),
+ print("\n%s [%d]" % (comm, pid))
id_keys = syscalls[comm][pid].keys()
- for id, val in sorted(syscalls[comm][pid].iteritems(), \
- key = lambda(k, v): (v, k), reverse = True):
- print " %-38s %10d\n" % (syscall_name(id), val),
+ for id, val in sorted(syscalls[comm][pid].items(), \
+ key = lambda kv: (kv[1], kv[0]), reverse =
True):
+ print(" %-38s %10d" % (syscall_name(id), val))
[tip:perf/core] perf script python: Add Python3 support to syscall-counts.py
Commit-ID: 1d1b0dbb859d175eb512a9f0e1ca7e44bd0192cd
Gitweb: https://git.kernel.org/tip/1d1b0dbb859d175eb512a9f0e1ca7e44bd0192cd
Author: Tony Jones
AuthorDate: Fri, 22 Feb 2019 15:06:17 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Mon, 25 Feb 2019 17:17:10 -0300
perf script python: Add Python3 support to syscall-counts.py
Support both Python2 and Python3 in the syscall-counts.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Link: http://lkml.kernel.org/r/2019030619.17887-14-to...@suse.de
Signed-off-by: Seeteena Thoufeek
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/syscall-counts.py | 18 ++
1 file changed, 10 insertions(+), 8 deletions(-)
diff --git a/tools/perf/scripts/python/syscall-counts.py
b/tools/perf/scripts/python/syscall-counts.py
index e66a7730aeb5..0ebd89cfd42c 100644
--- a/tools/perf/scripts/python/syscall-counts.py
+++ b/tools/perf/scripts/python/syscall-counts.py
@@ -5,6 +5,8 @@
# Displays system-wide system call totals, broken down by syscall.
# If a [comm] arg is specified, only syscalls called by [comm] are displayed.
+from __future__ import print_function
+
import os
import sys
@@ -28,7 +30,7 @@ if len(sys.argv) > 1:
syscalls = autodict()
def trace_begin():
- print "Press control+C to stop and show the summary"
+ print("Press control+C to stop and show the summary")
def trace_end():
print_syscall_totals()
@@ -51,14 +53,14 @@ def syscalls__sys_enter(event_name, context, common_cpu,
def print_syscall_totals():
if for_comm is not None:
- print "\nsyscall events for %s:\n\n" % (for_comm),
+ print("\nsyscall events for %s:\n" % (for_comm))
else:
- print "\nsyscall events:\n\n",
+ print("\nsyscall events:\n")
-print "%-40s %10s\n" % ("event", "count"),
-print "%-40s %10s\n" % ("", \
- "---"),
+print("%-40s %10s" % ("event", "count"))
+print("%-40s %10s" % ("",
+ "---"))
-for id, val in sorted(syscalls.iteritems(), key = lambda(k, v): (v, k), \
+for id, val in sorted(syscalls.items(), key = lambda kv: (kv[1], kv[0]), \
reverse = True):
- print "%-40s %10d\n" % (syscall_name(id), val),
+ print("%-40s %10d" % (syscall_name(id), val))
[tip:perf/core] perf script python: Add Python3 support to stat-cpi.py
Commit-ID: e985bf761db7646cebcd236249da08bd264069de
Gitweb: https://git.kernel.org/tip/e985bf761db7646cebcd236249da08bd264069de
Author: Tony Jones
AuthorDate: Fri, 22 Feb 2019 15:06:16 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Mon, 25 Feb 2019 17:17:07 -0300
perf script python: Add Python3 support to stat-cpi.py
Support both Python2 and Python3 in the stat-cpi.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Seeteena Thoufeek
Cc: Jiri Olsa
Link: http://lkml.kernel.org/r/2019030619.17887-13-to...@suse.de
Signed-off-by: Tony Jones
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/stat-cpi.py | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/tools/perf/scripts/python/stat-cpi.py
b/tools/perf/scripts/python/stat-cpi.py
index a81ad8835a74..01fa933ff3cf 100644
--- a/tools/perf/scripts/python/stat-cpi.py
+++ b/tools/perf/scripts/python/stat-cpi.py
@@ -1,5 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
+from __future__ import print_function
+
data= {}
times = []
threads = []
@@ -19,8 +21,8 @@ def store_key(time, cpu, thread):
threads.append(thread)
def store(time, event, cpu, thread, val, ena, run):
-#print "event %s cpu %d, thread %d, time %d, val %d, ena %d, run %d" % \
-# (event, cpu, thread, time, val, ena, run)
+#print("event %s cpu %d, thread %d, time %d, val %d, ena %d, run %d" %
+# (event, cpu, thread, time, val, ena, run))
store_key(time, cpu, thread)
key = get_key(time, event, cpu, thread)
@@ -58,7 +60,7 @@ def stat__interval(time):
if ins != 0:
cpi = cyc/float(ins)
-print "%15f: cpu %d, thread %d -> cpi %f (%d/%d)" %
(time/(float(10)), cpu, thread, cpi, cyc, ins)
+print("%15f: cpu %d, thread %d -> cpi %f (%d/%d)" %
(time/(float(10)), cpu, thread, cpi, cyc, ins))
def trace_end():
pass
@@ -74,4 +76,4 @@ def trace_end():
#if ins != 0:
#cpi = cyc/float(ins)
#
-#print "time %.9f, cpu %d, thread %d -> cpi %f" %
(time/(float(10)), cpu, thread, cpi)
+#print("time %.9f, cpu %d, thread %d -> cpi %f" %
(time/(float(10)), cpu, thread, cpi))
[tip:perf/core] perf script python: Add Python3 support to stackcollapse.py
Commit-ID: 6d22d9991cf37edfe861569e2433342ad56206a7
Gitweb: https://git.kernel.org/tip/6d22d9991cf37edfe861569e2433342ad56206a7
Author: Tony Jones
AuthorDate: Fri, 22 Feb 2019 15:06:15 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Mon, 25 Feb 2019 17:17:05 -0300
perf script python: Add Python3 support to stackcollapse.py
Support both Python2 and Python3 in the stackcollapse.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Cc: Paolo Bonzini
Link: http://lkml.kernel.org/r/2019030619.17887-12-to...@suse.de
Signed-off-by: Seeteena Thoufeek
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/stackcollapse.py | 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/tools/perf/scripts/python/stackcollapse.py
b/tools/perf/scripts/python/stackcollapse.py
index 1697b5e18c96..5e703efaddcc 100755
--- a/tools/perf/scripts/python/stackcollapse.py
+++ b/tools/perf/scripts/python/stackcollapse.py
@@ -19,6 +19,8 @@
# Written by Paolo Bonzini
# Based on Brendan Gregg's stackcollapse-perf.pl script.
+from __future__ import print_function
+
import os
import sys
from collections import defaultdict
@@ -120,7 +122,6 @@ def process_event(param_dict):
lines[stack_string] = lines[stack_string] + 1
def trace_end():
-list = lines.keys()
-list.sort()
+list = sorted(lines)
for stack in list:
-print "%s %d" % (stack, lines[stack])
+print("%s %d" % (stack, lines[stack]))
[tip:perf/core] perf script python: Add Python3 support to net_dropmonitor.py
Commit-ID: 8c42b9600e561666233b9c557a5209d0dc853ba1
Gitweb: https://git.kernel.org/tip/8c42b9600e561666233b9c557a5209d0dc853ba1
Author: Tony Jones
AuthorDate: Fri, 22 Feb 2019 15:06:12 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Mon, 25 Feb 2019 17:16:55 -0300
perf script python: Add Python3 support to net_dropmonitor.py
Support both Python2 and Python3 in the net_dropmonitor.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Acked-by: Neil Horman
Link: http://lkml.kernel.org/r/2019030619.17887-9-to...@suse.de
Signed-off-by: Seeteena Thoufeek
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/net_dropmonitor.py | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/tools/perf/scripts/python/net_dropmonitor.py
b/tools/perf/scripts/python/net_dropmonitor.py
index a150164b44a3..212557a02c50 100755
--- a/tools/perf/scripts/python/net_dropmonitor.py
+++ b/tools/perf/scripts/python/net_dropmonitor.py
@@ -1,6 +1,8 @@
# Monitor the system for dropped packets and proudce a report of drop
locations and counts
# SPDX-License-Identifier: GPL-2.0
+from __future__ import print_function
+
import os
import sys
@@ -50,19 +52,19 @@ def get_sym(sloc):
return (None, 0)
def print_drop_table():
- print "%25s %25s %25s" % ("LOCATION", "OFFSET", "COUNT")
+ print("%25s %25s %25s" % ("LOCATION", "OFFSET", "COUNT"))
for i in drop_log.keys():
(sym, off) = get_sym(i)
if sym == None:
sym = i
- print "%25s %25s %25s" % (sym, off, drop_log[i])
+ print("%25s %25s %25s" % (sym, off, drop_log[i]))
def trace_begin():
- print "Starting trace (Ctrl-C to dump results)"
+ print("Starting trace (Ctrl-C to dump results)")
def trace_end():
- print "Gathering kallsyms data"
+ print("Gathering kallsyms data")
get_kallsyms_table()
print_drop_table()
[tip:perf/core] perf script python: Add Python3 support to sctop.py
Commit-ID: ee75a896ae535d4219a82cc361be96394536f3ba
Gitweb: https://git.kernel.org/tip/ee75a896ae535d4219a82cc361be96394536f3ba
Author: Tony Jones
AuthorDate: Fri, 22 Feb 2019 15:06:14 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Mon, 25 Feb 2019 17:17:03 -0300
perf script python: Add Python3 support to sctop.py
Support both Python2 and Python3 in the sctop.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Cc: Tom Zanussi
Link: http://lkml.kernel.org/r/2019030619.17887-11-to...@suse.de
Signed-off-by: Seeteena Thoufeek
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/sctop.py | 24
1 file changed, 16 insertions(+), 8 deletions(-)
diff --git a/tools/perf/scripts/python/sctop.py
b/tools/perf/scripts/python/sctop.py
index 61621b93affb..987ffae7c8ca 100644
--- a/tools/perf/scripts/python/sctop.py
+++ b/tools/perf/scripts/python/sctop.py
@@ -8,7 +8,14 @@
# will be refreshed every [interval] seconds. The default interval is
# 3 seconds.
-import os, sys, thread, time
+from __future__ import print_function
+
+import os, sys, time
+
+try:
+import thread
+except ImportError:
+import _thread as thread
sys.path.append(os.environ['PERF_EXEC_PATH'] + \
'/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
@@ -62,18 +69,19 @@ def print_syscall_totals(interval):
while 1:
clear_term()
if for_comm is not None:
- print "\nsyscall events for %s:\n\n" % (for_comm),
+ print("\nsyscall events for %s:\n" % (for_comm))
else:
- print "\nsyscall events:\n\n",
+ print("\nsyscall events:\n")
- print "%-40s %10s\n" % ("event", "count"),
- print "%-40s %10s\n" %
("", \
-"--"),
+ print("%-40s %10s" % ("event", "count"))
+ print("%-40s %10s" %
+("",
+"--"))
- for id, val in sorted(syscalls.iteritems(), key = lambda(k, v):
(v, k), \
+ for id, val in sorted(syscalls.items(), key = lambda kv:
(kv[1], kv[0]), \
reverse = True):
try:
- print "%-40s %10d\n" % (syscall_name(id), val),
+ print("%-40s %10d" % (syscall_name(id), val))
except TypeError:
pass
syscalls.clear()
[tip:perf/core] perf script python: Add Python3 support to powerpc-hcalls.py
Commit-ID: 118af5bf799bd1876c3999766d1d2f845d45f019
Gitweb: https://git.kernel.org/tip/118af5bf799bd1876c3999766d1d2f845d45f019
Author: Tony Jones
AuthorDate: Fri, 22 Feb 2019 15:06:13 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Mon, 25 Feb 2019 17:16:57 -0300
perf script python: Add Python3 support to powerpc-hcalls.py
Support both Python2 and Python3 in the powerpc-hcalls.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Cc: Ravi Bangoria
Link: http://lkml.kernel.org/r/2019030619.17887-10-to...@suse.de
Signed-off-by: Seeteena Thoufeek
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/powerpc-hcalls.py | 18 ++
1 file changed, 10 insertions(+), 8 deletions(-)
diff --git a/tools/perf/scripts/python/powerpc-hcalls.py
b/tools/perf/scripts/python/powerpc-hcalls.py
index 00e0e7476e55..8b78dc790adb 100644
--- a/tools/perf/scripts/python/powerpc-hcalls.py
+++ b/tools/perf/scripts/python/powerpc-hcalls.py
@@ -4,6 +4,8 @@
#
# Hypervisor call statisics
+from __future__ import print_function
+
import os
import sys
@@ -149,7 +151,7 @@ hcall_table = {
}
def hcall_table_lookup(opcode):
- if (hcall_table.has_key(opcode)):
+ if (opcode in hcall_table):
return hcall_table[opcode]
else:
return opcode
@@ -157,8 +159,8 @@ def hcall_table_lookup(opcode):
print_ptrn = '%-28s%10s%10s%10s%10s'
def trace_end():
- print print_ptrn % ('hcall', 'count', 'min(ns)', 'max(ns)', 'avg(ns)')
- print '-' * 68
+ print(print_ptrn % ('hcall', 'count', 'min(ns)', 'max(ns)', 'avg(ns)'))
+ print('-' * 68)
for opcode in output:
h_name = hcall_table_lookup(opcode)
time = output[opcode]['time']
@@ -166,14 +168,14 @@ def trace_end():
min_t = output[opcode]['min']
max_t = output[opcode]['max']
- print print_ptrn % (h_name, cnt, min_t, max_t, time/cnt)
+ print(print_ptrn % (h_name, cnt, min_t, max_t, time//cnt))
def powerpc__hcall_exit(name, context, cpu, sec, nsec, pid, comm, callchain,
opcode, retval):
- if (d_enter.has_key(cpu) and d_enter[cpu].has_key(opcode)):
+ if (cpu in d_enter and opcode in d_enter[cpu]):
diff = nsecs(sec, nsec) - d_enter[cpu][opcode]
- if (output.has_key(opcode)):
+ if (opcode in output):
output[opcode]['time'] += diff
output[opcode]['cnt'] += 1
if (output[opcode]['min'] > diff):
@@ -190,11 +192,11 @@ def powerpc__hcall_exit(name, context, cpu, sec, nsec,
pid, comm, callchain,
del d_enter[cpu][opcode]
# else:
-# print "Can't find matching hcall_enter event. Ignoring sample"
+# print("Can't find matching hcall_enter event. Ignoring sample")
def powerpc__hcall_entry(event_name, context, cpu, sec, nsec, pid, comm,
callchain, opcode):
- if (d_enter.has_key(cpu)):
+ if (cpu in d_enter):
d_enter[cpu][opcode] = nsecs(sec, nsec)
else:
d_enter[cpu] = {opcode: nsecs(sec, nsec)}
[tip:perf/core] perf script python: Add Python3 support to mem-phys-addr.py
Commit-ID: e4d053ddb4c48cbde27b4c5edd3cc8f957684e4f
Gitweb: https://git.kernel.org/tip/e4d053ddb4c48cbde27b4c5edd3cc8f957684e4f
Author: Tony Jones
AuthorDate: Fri, 22 Feb 2019 15:06:11 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Mon, 25 Feb 2019 17:16:51 -0300
perf script python: Add Python3 support to mem-phys-addr.py
Support both Python2 and Python3 in the mem-phys-addr.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Link: http://lkml.kernel.org/r/2019030619.17887-8-to...@suse.de
Signed-off-by: Seeteena Thoufeek
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/mem-phys-addr.py | 24 ++--
1 file changed, 14 insertions(+), 10 deletions(-)
diff --git a/tools/perf/scripts/python/mem-phys-addr.py
b/tools/perf/scripts/python/mem-phys-addr.py
index ebee2c5ae496..fb0bbcbfa0f0 100644
--- a/tools/perf/scripts/python/mem-phys-addr.py
+++ b/tools/perf/scripts/python/mem-phys-addr.py
@@ -4,6 +4,8 @@
# Copyright (c) 2018, Intel Corporation.
from __future__ import division
+from __future__ import print_function
+
import os
import sys
import struct
@@ -31,21 +33,23 @@ def parse_iomem():
for i, j in enumerate(f):
m = re.split('-|:',j,2)
if m[2].strip() == 'System RAM':
- system_ram.append(long(m[0], 16))
- system_ram.append(long(m[1], 16))
+ system_ram.append(int(m[0], 16))
+ system_ram.append(int(m[1], 16))
if m[2].strip() == 'Persistent Memory':
- pmem.append(long(m[0], 16))
- pmem.append(long(m[1], 16))
+ pmem.append(int(m[0], 16))
+ pmem.append(int(m[1], 16))
def print_memory_type():
- print "Event: %s" % (event_name)
- print "%-40s %10s %10s\n" % ("Memory type", "count", "percentage"),
- print "%-40s %10s %10s\n" %
("", \
+ print("Event: %s" % (event_name))
+ print("%-40s %10s %10s\n" % ("Memory type", "count", "percentage"),
end='')
+ print("%-40s %10s %10s\n" %
("",
"---", "---"),
+end='');
total = sum(load_mem_type_cnt.values())
for mem_type, count in sorted(load_mem_type_cnt.most_common(), \
- key = lambda(k, v): (v, k), reverse =
True):
- print "%-40s %10d %10.1f%%\n" % (mem_type, count, 100 * count
/ total),
+ key = lambda kv: (kv[1], kv[0]),
reverse = True):
+ print("%-40s %10d %10.1f%%\n" % (mem_type, count, 100 * count
/ total),
+end='')
def trace_begin():
parse_iomem()
@@ -80,7 +84,7 @@ def find_memory_type(phys_addr):
f.seek(0, 0)
for j in f:
m = re.split('-|:',j,2)
- if long(m[0], 16) <= phys_addr <= long(m[1], 16):
+ if int(m[0], 16) <= phys_addr <= int(m[1], 16):
return m[2]
return "N/A"
[tip:perf/core] perf script python: Add Python3 support to failed-syscalls-by-pid.py
Commit-ID: 9b2700efc57f46fe63beee5f64fcfe2746936b4e
Gitweb: https://git.kernel.org/tip/9b2700efc57f46fe63beee5f64fcfe2746936b4e
Author: Tony Jones
AuthorDate: Fri, 22 Feb 2019 15:06:08 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Mon, 25 Feb 2019 17:16:48 -0300
perf script python: Add Python3 support to failed-syscalls-by-pid.py
Support both Python2 and Python3 in the failed-syscalls-by-pid.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Cc: Tom Zanussi
Link: http://lkml.kernel.org/r/2019030619.17887-5-to...@suse.de
Signed-off-by: Seeteena Thoufeek
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/failed-syscalls-by-pid.py | 21 +++--
1 file changed, 11 insertions(+), 10 deletions(-)
diff --git a/tools/perf/scripts/python/failed-syscalls-by-pid.py
b/tools/perf/scripts/python/failed-syscalls-by-pid.py
index cafeff3d74db..3648e8b986ec 100644
--- a/tools/perf/scripts/python/failed-syscalls-by-pid.py
+++ b/tools/perf/scripts/python/failed-syscalls-by-pid.py
@@ -5,6 +5,8 @@
# Displays system-wide failed system call totals, broken down by pid.
# If a [comm] arg is specified, only syscalls called by [comm] are displayed.
+from __future__ import print_function
+
import os
import sys
@@ -32,7 +34,7 @@ if len(sys.argv) > 1:
syscalls = autodict()
def trace_begin():
- print "Press control+C to stop and show the summary"
+ print("Press control+C to stop and show the summary")
def trace_end():
print_error_totals()
@@ -57,22 +59,21 @@ def syscalls__sys_exit(event_name, context, common_cpu,
def print_error_totals():
if for_comm is not None:
- print "\nsyscall errors for %s:\n\n" % (for_comm),
+ print("\nsyscall errors for %s:\n" % (for_comm))
else:
- print "\nsyscall errors:\n\n",
+ print("\nsyscall errors:\n")
-print "%-30s %10s\n" % ("comm [pid]", "count"),
-print "%-30s %10s\n" % ("--", \
- "--"),
+print("%-30s %10s" % ("comm [pid]", "count"))
+print("%-30s %10s" % ("--", "--"))
comm_keys = syscalls.keys()
for comm in comm_keys:
pid_keys = syscalls[comm].keys()
for pid in pid_keys:
- print "\n%s [%d]\n" % (comm, pid),
+ print("\n%s [%d]" % (comm, pid))
id_keys = syscalls[comm][pid].keys()
for id in id_keys:
- print " syscall: %-16s\n" % syscall_name(id),
+ print(" syscall: %-16s" % syscall_name(id))
ret_keys = syscalls[comm][pid][id].keys()
- for ret, val in
sorted(syscalls[comm][pid][id].iteritems(), key = lambda(k, v): (v, k),
reverse = True):
- print "err = %-20s %10d\n" %
(strerror(ret), val),
+ for ret, val in
sorted(syscalls[comm][pid][id].items(), key = lambda kv: (kv[1], kv[0]),
reverse = True):
+ print("err = %-20s %10d" %
(strerror(ret), val))
[tip:perf/core] perf script python: Add Python3 support to netdev-times.py
Commit-ID: 02b03ec383e0c79d73aa4b402b3427a8b490ef9f
Gitweb: https://git.kernel.org/tip/02b03ec383e0c79d73aa4b402b3427a8b490ef9f
Author: Tony Jones
AuthorDate: Fri, 22 Feb 2019 15:06:05 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Mon, 25 Feb 2019 17:16:42 -0300
perf script python: Add Python3 support to netdev-times.py
Support both Python2 and Python3 in the netdev-times.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2
version is now v2.6.
Signed-off-by: Tony Jones
Cc: Sanagi Koki
Link: http://lkml.kernel.org/r/2019030619.17887-2-to...@suse.de
Signed-off-by: Seeteena Thoufeek
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/netdev-times.py | 82 ---
1 file changed, 42 insertions(+), 40 deletions(-)
diff --git a/tools/perf/scripts/python/netdev-times.py
b/tools/perf/scripts/python/netdev-times.py
index 9b2050f778f1..267bda49325d 100644
--- a/tools/perf/scripts/python/netdev-times.py
+++ b/tools/perf/scripts/python/netdev-times.py
@@ -8,6 +8,8 @@
# dev=: show only thing related to specified device
# debug: work with debug mode. It shows buffer status.
+from __future__ import print_function
+
import os
import sys
@@ -17,6 +19,7 @@ sys.path.append(os.environ['PERF_EXEC_PATH'] + \
from perf_trace_context import *
from Core import *
from Util import *
+from functools import cmp_to_key
all_event_list = []; # insert all tracepoint event related with this script
irq_dic = {}; # key is cpu and value is a list which stacks irqs
@@ -61,12 +64,12 @@ def diff_msec(src, dst):
def print_transmit(hunk):
if dev != 0 and hunk['dev'].find(dev) < 0:
return
- print "%7s %5d %6d.%06dsec %12.3fmsec %12.3fmsec" % \
+ print("%7s %5d %6d.%06dsec %12.3fmsec %12.3fmsec" %
(hunk['dev'], hunk['len'],
nsecs_secs(hunk['queue_t']),
nsecs_nsecs(hunk['queue_t'])/1000,
diff_msec(hunk['queue_t'], hunk['xmit_t']),
- diff_msec(hunk['xmit_t'], hunk['free_t']))
+ diff_msec(hunk['xmit_t'], hunk['free_t'])))
# Format for displaying rx packet processing
PF_IRQ_ENTRY= " irq_entry(+%.3fmsec irq=%d:%s)"
@@ -98,55 +101,55 @@ def print_receive(hunk):
if show_hunk == 0:
return
- print "%d.%06dsec cpu=%d" % \
- (nsecs_secs(base_t), nsecs_nsecs(base_t)/1000, cpu)
+ print("%d.%06dsec cpu=%d" %
+ (nsecs_secs(base_t), nsecs_nsecs(base_t)/1000, cpu))
for i in range(len(irq_list)):
- print PF_IRQ_ENTRY % \
+ print(PF_IRQ_ENTRY %
(diff_msec(base_t, irq_list[i]['irq_ent_t']),
- irq_list[i]['irq'], irq_list[i]['name'])
- print PF_JOINT
+ irq_list[i]['irq'], irq_list[i]['name']))
+ print(PF_JOINT)
irq_event_list = irq_list[i]['event_list']
for j in range(len(irq_event_list)):
irq_event = irq_event_list[j]
if irq_event['event'] == 'netif_rx':
- print PF_NET_RX % \
+ print(PF_NET_RX %
(diff_msec(base_t, irq_event['time']),
- irq_event['skbaddr'])
- print PF_JOINT
- print PF_SOFT_ENTRY % \
- diff_msec(base_t, hunk['sirq_ent_t'])
- print PF_JOINT
+ irq_event['skbaddr']))
+ print(PF_JOINT)
+ print(PF_SOFT_ENTRY %
+ diff_msec(base_t, hunk['sirq_ent_t']))
+ print(PF_JOINT)
event_list = hunk['event_list']
for i in range(len(event_list)):
event = event_list[i]
if event['event_name'] == 'napi_poll':
- print PF_NAPI_POLL % \
- (diff_msec(base_t, event['event_t']), event['dev'])
+ print(PF_NAPI_POLL %
+ (diff_msec(base_t, event['event_t']), event['dev']))
if i == len(event_list) - 1:
- print ""
+ print("")
else:
- print PF_JOINT
+ print(PF_JOINT)
else:
- print PF_NET_RECV % \
+ print(PF_NET_RECV %
(diff_msec(base_t, event['event_t']),
event['skbaddr'],
- event['len'])
+
[PATCH] tools lib traceevent: Fix buffer overflow in arg_eval
Fix buffer overflow observed when running perf test.
The overflow is when trying to evaluate "1ULL << (64 - 1)" which
is resulting in -9223372036854775808 which overflows the 20 character
buffer.
If is possible this bug has been reported before but I still don't
see any fix checked in:
See: https://www.spinics.net/lists/linux-perf-users/msg07714.html
Cc: Arnaldo Carvalho de Melo
Cc: linux-perf-us...@vger.kernel.org
Cc: Steven Rostedt
Signed-off-by: Tony Jones
---
tools/lib/traceevent/event-parse.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/lib/traceevent/event-parse.c
b/tools/lib/traceevent/event-parse.c
index abd4fa5d3088..87494c7c619d 100644
--- a/tools/lib/traceevent/event-parse.c
+++ b/tools/lib/traceevent/event-parse.c
@@ -2457,7 +2457,7 @@ static int arg_num_eval(struct tep_print_arg *arg, long
long *val)
static char *arg_eval (struct tep_print_arg *arg)
{
long long val;
- static char buf[20];
+ static char buf[24];
switch (arg->type) {
case TEP_PRINT_ATOM:
--
2.20.1
Re: [PATCH 02/15] perf script python: add Python3 support to check-perf-trace.py
On 2/25/19 6:05 AM, Arnaldo Carvalho de Melo wrote: > Em Fri, Feb 22, 2019 at 03:06:06PM -0800, Tony Jones escreveu: >> Support both Python 2 and Python 3 in the check-perf-trace.py script. >> >> There may be differences in the ordering of output lines due to >> differences in dictionary ordering etc. However the format within lines >> should be unchanged. >> >> Also correct inconsistent indentation which was being flagged by Python3. > > When we have "Also" in a patch, usually it need to be split :-) The indentation changes were required for Python3. So I viewed them as part of the change to support Python3 ;-) However, splitting them is no problem.I'll resubmit.
Re: [PATCH 15/15] perf script python: add Python3 support to sql scripts
On 2/25/19 6:54 AM, Arnaldo Carvalho de Melo wrote: > Em Mon, Feb 25, 2019 at 11:51:19AM -0300, Arnaldo Carvalho de Melo escreveu: >> Em Fri, Feb 22, 2019 at 03:06:19PM -0800, Tony Jones escreveu: >>> Support both Python2 and Python3 in the exported-sql-viewer.py, >>> export-to-postgresql.py and export-to-sqlite.py scripts > >>> @@ -234,12 +236,17 @@ perf_db_export_mode = True >>> perf_db_export_calls = False >>> perf_db_export_callchains = False >>> >>> +def printerr(*args, **kw_args): >>> +print(*args, file=sys.stderr, **kw_args) >>> + >>> +def printdate(*args, **kw_args): >>> +print(datetime.datetime.today(), *args, sep=' ', **kw_args) >> >> So this one introduces inconsistencies in indentation, i.e. the above >> two routines don't use tabs and one uses 4 spaces while the other uses 8 >> spaces :-\ >> >> I'm converting them to use tabs, like the rest of this script, ok? > > But it is not applying, Adrian did work recently here, can you please > address this and resubmit? I was using perf/perf/core. I'll take a look. Also, I agree we should be consistent on indentation.
Re: [PATCH 06/15] perf script python: add Python3 support to intel-pt-events.py
On 2/25/19 6:03 AM, Arnaldo Carvalho de Melo wrote: > Em Mon, Feb 25, 2019 at 09:07:03AM +0200, Adrian Hunter escreveu: >> On 23/02/19 1:06 AM, Tony Jones wrote: >>> Support both Python2 and Python3 in the intel-pt-events.py script >>> >>> There may be differences in the ordering of output lines due to >>> differences in dictionary ordering etc. However the format within lines >>> should be unchanged. >>> >>> Fix space/tab inconsistency as python3 enforces consistency. >> >> It would be better for the white space changes to be a separate patch. >> >> But I am not in favour of using spaces instead of tabs because it is the >> opposite of what we tend to do with C. >> >> Arnaldo, can you provide a guideline on this? > > Yeah, please separate things, this would help me now to pick the > uncontroversial part while we discuss the other. > > And I'm in favour of being consistent with what we do for C code, i.e. > use TABs. > > I'm now processing the other patches. I'll split and resubmit. Also, FYI, PEP8 states "Spaces are the preferred indentation method".
Re: [PATCH 00/15] perf script python: add Python3 support
On 2/22/19 3:06 PM, Tony Jones wrote: > The following patches add Python3 support to the remainder of > the scripts under scripts/python. I should possibly have labeled this V3. Seeteena had submitted prior patches but they had issues and since I already had (better tested) patches in OpenSUSE and SLES that I was about to submit, we decided I would submit those patches under a joint signed-off-by, which is what I've done here. Apologies for any confusion. Tony
[PATCH 08/15] perf script python: add Python3 support to net_dropmonitor.py
Support both Python2 and Python3 in the net_dropmonitor.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Neil Horman
---
tools/perf/scripts/python/net_dropmonitor.py | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/tools/perf/scripts/python/net_dropmonitor.py
b/tools/perf/scripts/python/net_dropmonitor.py
index a150164b44a3..212557a02c50 100755
--- a/tools/perf/scripts/python/net_dropmonitor.py
+++ b/tools/perf/scripts/python/net_dropmonitor.py
@@ -1,6 +1,8 @@
# Monitor the system for dropped packets and proudce a report of drop
locations and counts
# SPDX-License-Identifier: GPL-2.0
+from __future__ import print_function
+
import os
import sys
@@ -50,19 +52,19 @@ def get_sym(sloc):
return (None, 0)
def print_drop_table():
- print "%25s %25s %25s" % ("LOCATION", "OFFSET", "COUNT")
+ print("%25s %25s %25s" % ("LOCATION", "OFFSET", "COUNT"))
for i in drop_log.keys():
(sym, off) = get_sym(i)
if sym == None:
sym = i
- print "%25s %25s %25s" % (sym, off, drop_log[i])
+ print("%25s %25s %25s" % (sym, off, drop_log[i]))
def trace_begin():
- print "Starting trace (Ctrl-C to dump results)"
+ print("Starting trace (Ctrl-C to dump results)")
def trace_end():
- print "Gathering kallsyms data"
+ print("Gathering kallsyms data")
get_kallsyms_table()
print_drop_table()
--
2.20.1
[PATCH 09/15] perf script python: add Python3 support to powerpc-hcalls.py
Support both Python2 and Python3 in the powerpc-hcalls.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Ravi Bangoria
---
tools/perf/scripts/python/powerpc-hcalls.py | 18 ++
1 file changed, 10 insertions(+), 8 deletions(-)
diff --git a/tools/perf/scripts/python/powerpc-hcalls.py
b/tools/perf/scripts/python/powerpc-hcalls.py
index 00e0e7476e55..8b78dc790adb 100644
--- a/tools/perf/scripts/python/powerpc-hcalls.py
+++ b/tools/perf/scripts/python/powerpc-hcalls.py
@@ -4,6 +4,8 @@
#
# Hypervisor call statisics
+from __future__ import print_function
+
import os
import sys
@@ -149,7 +151,7 @@ hcall_table = {
}
def hcall_table_lookup(opcode):
- if (hcall_table.has_key(opcode)):
+ if (opcode in hcall_table):
return hcall_table[opcode]
else:
return opcode
@@ -157,8 +159,8 @@ def hcall_table_lookup(opcode):
print_ptrn = '%-28s%10s%10s%10s%10s'
def trace_end():
- print print_ptrn % ('hcall', 'count', 'min(ns)', 'max(ns)', 'avg(ns)')
- print '-' * 68
+ print(print_ptrn % ('hcall', 'count', 'min(ns)', 'max(ns)', 'avg(ns)'))
+ print('-' * 68)
for opcode in output:
h_name = hcall_table_lookup(opcode)
time = output[opcode]['time']
@@ -166,14 +168,14 @@ def trace_end():
min_t = output[opcode]['min']
max_t = output[opcode]['max']
- print print_ptrn % (h_name, cnt, min_t, max_t, time/cnt)
+ print(print_ptrn % (h_name, cnt, min_t, max_t, time//cnt))
def powerpc__hcall_exit(name, context, cpu, sec, nsec, pid, comm, callchain,
opcode, retval):
- if (d_enter.has_key(cpu) and d_enter[cpu].has_key(opcode)):
+ if (cpu in d_enter and opcode in d_enter[cpu]):
diff = nsecs(sec, nsec) - d_enter[cpu][opcode]
- if (output.has_key(opcode)):
+ if (opcode in output):
output[opcode]['time'] += diff
output[opcode]['cnt'] += 1
if (output[opcode]['min'] > diff):
@@ -190,11 +192,11 @@ def powerpc__hcall_exit(name, context, cpu, sec, nsec,
pid, comm, callchain,
del d_enter[cpu][opcode]
# else:
-# print "Can't find matching hcall_enter event. Ignoring sample"
+# print("Can't find matching hcall_enter event. Ignoring sample")
def powerpc__hcall_entry(event_name, context, cpu, sec, nsec, pid, comm,
callchain, opcode):
- if (d_enter.has_key(cpu)):
+ if (cpu in d_enter):
d_enter[cpu][opcode] = nsecs(sec, nsec)
else:
d_enter[cpu] = {opcode: nsecs(sec, nsec)}
--
2.20.1
[PATCH 14/15] perf script python: add Python3 support to syscall-counts-by-pid.py
Support both Python2 and Python3 in the syscall-counts-by-pid.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
---
.../scripts/python/syscall-counts-by-pid.py | 22 ++-
1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/tools/perf/scripts/python/syscall-counts-by-pid.py
b/tools/perf/scripts/python/syscall-counts-by-pid.py
index daf314cc5dd3..42782487b0e9 100644
--- a/tools/perf/scripts/python/syscall-counts-by-pid.py
+++ b/tools/perf/scripts/python/syscall-counts-by-pid.py
@@ -5,6 +5,8 @@
# Displays system-wide system call totals, broken down by syscall.
# If a [comm] arg is specified, only syscalls called by [comm] are displayed.
+from __future__ import print_function
+
import os, sys
sys.path.append(os.environ['PERF_EXEC_PATH'] + \
@@ -31,7 +33,7 @@ if len(sys.argv) > 1:
syscalls = autodict()
def trace_begin():
- print "Press control+C to stop and show the summary"
+ print("Press control+C to stop and show the summary")
def trace_end():
print_syscall_totals()
@@ -55,20 +57,20 @@ def syscalls__sys_enter(event_name, context, common_cpu,
def print_syscall_totals():
if for_comm is not None:
- print "\nsyscall events for %s:\n\n" % (for_comm),
+ print("\nsyscall events for %s:\n" % (for_comm))
else:
- print "\nsyscall events by comm/pid:\n\n",
+ print("\nsyscall events by comm/pid:\n")
-print "%-40s %10s\n" % ("comm [pid]/syscalls", "count"),
-print "%-40s %10s\n" % ("", \
- "--"),
+print("%-40s %10s" % ("comm [pid]/syscalls", "count"))
+print("%-40s %10s" % ("",
+"--"))
comm_keys = syscalls.keys()
for comm in comm_keys:
pid_keys = syscalls[comm].keys()
for pid in pid_keys:
- print "\n%s [%d]\n" % (comm, pid),
+ print("\n%s [%d]" % (comm, pid))
id_keys = syscalls[comm][pid].keys()
- for id, val in sorted(syscalls[comm][pid].iteritems(), \
- key = lambda(k, v): (v, k), reverse = True):
- print " %-38s %10d\n" % (syscall_name(id), val),
+ for id, val in sorted(syscalls[comm][pid].items(), \
+ key = lambda kv: (kv[1], kv[0]), reverse =
True):
+ print(" %-38s %10d" % (syscall_name(id), val))
--
2.20.1
[PATCH 15/15] perf script python: add Python3 support to sql scripts
Support both Python2 and Python3 in the exported-sql-viewer.py,
export-to-postgresql.py and export-to-sqlite.py scripts
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
---
.../scripts/python/export-to-postgresql.py| 31 ++
tools/perf/scripts/python/export-to-sqlite.py | 26 ++---
.../scripts/python/exported-sql-viewer.py | 56 ---
3 files changed, 71 insertions(+), 42 deletions(-)
diff --git a/tools/perf/scripts/python/export-to-postgresql.py
b/tools/perf/scripts/python/export-to-postgresql.py
index 30130213da7e..baf972680dc7 100644
--- a/tools/perf/scripts/python/export-to-postgresql.py
+++ b/tools/perf/scripts/python/export-to-postgresql.py
@@ -10,6 +10,8 @@
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
+from __future__ import print_function
+
import os
import sys
import struct
@@ -234,12 +236,17 @@ perf_db_export_mode = True
perf_db_export_calls = False
perf_db_export_callchains = False
+def printerr(*args, **kw_args):
+print(*args, file=sys.stderr, **kw_args)
+
+def printdate(*args, **kw_args):
+print(datetime.datetime.today(), *args, sep=' ', **kw_args)
def usage():
- print >> sys.stderr, "Usage is: export-to-postgresql.py
[] [] []"
- print >> sys.stderr, "where:columns 'all' or 'branches'"
- print >> sys.stderr, " calls 'calls' => create calls
and call_paths table"
- print >> sys.stderr, " callchains 'callchains' => create
call_paths table"
+ printerr("Usage is: export-to-postgresql.py []
[] []")
+ printerr("where:columns 'all' or 'branches'")
+ printerr(" calls 'calls' => create calls and
call_paths table")
+ printerr(" callchains 'callchains' => create
call_paths table")
raise Exception("Too few arguments")
if (len(sys.argv) < 2):
@@ -273,7 +280,7 @@ def do_query(q, s):
return
raise Exception("Query failed: " + q.lastError().text())
-print datetime.datetime.today(), "Creating database..."
+printdate("Creating database...")
db = QSqlDatabase.addDatabase('QPSQL')
query = QSqlQuery(db)
@@ -564,7 +571,7 @@ if perf_db_export_calls:
call_file = open_output_file("call_table.bin")
def trace_begin():
- print datetime.datetime.today(), "Writing to intermediate files..."
+ printdate("Writing to intermediate files...")
# id == 0 means unknown. It is easier to create records for them than
replace the zeroes with NULLs
evsel_table(0, "unknown")
machine_table(0, 0, "unknown")
@@ -579,7 +586,7 @@ def trace_begin():
unhandled_count = 0
def trace_end():
- print datetime.datetime.today(), "Copying to database..."
+ printdate("Copying to database...")
copy_output_file(evsel_file,"selected_events")
copy_output_file(machine_file, "machines")
copy_output_file(thread_file, "threads")
@@ -594,7 +601,7 @@ def trace_end():
if perf_db_export_calls:
copy_output_file(call_file, "calls")
- print datetime.datetime.today(), "Removing intermediate files..."
+ printdate("Removing intermediate files...")
remove_output_file(evsel_file)
remove_output_file(machine_file)
remove_output_file(thread_file)
@@ -609,7 +616,7 @@ def trace_end():
if perf_db_export_calls:
remove_output_file(call_file)
os.rmdir(output_dir_name)
- print datetime.datetime.today(), "Adding primary keys"
+ printdate("Adding primary keys")
do_query(query, 'ALTER TABLE selected_events ADD PRIMARY KEY (id)')
do_query(query, 'ALTER TABLE machinesADD PRIMARY KEY (id)')
do_query(query, 'ALTER TABLE threads ADD PRIMARY KEY (id)')
@@ -624,7 +631,7 @@ def trace_end():
if perf_db_export_calls:
do_query(query, 'ALTER TABLE calls ADD PRIMARY KEY
(id)')
- print datetime.datetime.today(), "Adding foreign keys"
+ printdate("Adding foreign keys")
do_query(query, 'ALTER TABLE threads '
'ADD CONSTRAINT machinefk FOREIGN KEY
(machine_id) REFERENCES machines (id),'
'ADD CONSTRA
[PATCH 02/15] perf script python: add Python3 support to check-perf-trace.py
Support both Python 2 and Python 3 in the check-perf-trace.py script.
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
Also correct inconsistent indentation which was being flagged by Python3.
The use of from __future__ implies the minimum supported version of
Python2 is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Tom Zanussi
---
tools/perf/scripts/python/check-perf-trace.py | 68 ++-
1 file changed, 36 insertions(+), 32 deletions(-)
diff --git a/tools/perf/scripts/python/check-perf-trace.py
b/tools/perf/scripts/python/check-perf-trace.py
index 334599c6032c..2851cf0e6b4b 100644
--- a/tools/perf/scripts/python/check-perf-trace.py
+++ b/tools/perf/scripts/python/check-perf-trace.py
@@ -7,11 +7,13 @@
# events, etc. Basically, if this script runs successfully and
# displays expected results, Python scripting support should be ok.
+from __future__ import print_function
+
import os
import sys
-sys.path.append(os.environ['PERF_EXEC_PATH'] + \
- '/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
+sys.path.append(os.environ['PERF_EXEC_PATH'] +
+'/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
from Core import *
from perf_trace_context import *
@@ -19,37 +21,38 @@ from perf_trace_context import *
unhandled = autodict()
def trace_begin():
- print "trace_begin"
- pass
+print("trace_begin")
+pass
def trace_end():
-print_unhandled()
+print_unhandled()
def irq__softirq_entry(event_name, context, common_cpu,
- common_secs, common_nsecs, common_pid, common_comm,
- common_callchain, vec):
- print_header(event_name, common_cpu, common_secs, common_nsecs,
- common_pid, common_comm)
+common_secs, common_nsecs, common_pid, common_comm,
+common_callchain, vec):
+
+print_header(event_name, common_cpu, common_secs, common_nsecs,
+common_pid, common_comm)
-print_uncommon(context)
+print_uncommon(context)
- print "vec=%s\n" % \
- (symbol_str("irq__softirq_entry", "vec", vec)),
+print("vec=%s" %
+(symbol_str("irq__softirq_entry", "vec", vec)))
def kmem__kmalloc(event_name, context, common_cpu,
- common_secs, common_nsecs, common_pid, common_comm,
- common_callchain, call_site, ptr, bytes_req, bytes_alloc,
- gfp_flags):
- print_header(event_name, common_cpu, common_secs, common_nsecs,
- common_pid, common_comm)
+common_secs, common_nsecs, common_pid, common_comm,
+common_callchain, call_site, ptr, bytes_req, bytes_alloc,
+gfp_flags):
-print_uncommon(context)
+print_header(event_name, common_cpu, common_secs, common_nsecs,
+common_pid, common_comm)
- print "call_site=%u, ptr=%u, bytes_req=%u, " \
- "bytes_alloc=%u, gfp_flags=%s\n" % \
- (call_site, ptr, bytes_req, bytes_alloc,
+print_uncommon(context)
- flag_str("kmem__kmalloc", "gfp_flags", gfp_flags)),
+print("call_site=%u, ptr=%u, bytes_req=%u, "
+"bytes_alloc=%u, gfp_flags=%s\n" %
+(call_site, ptr, bytes_req, bytes_alloc,
+flag_str("kmem__kmalloc", "gfp_flags", gfp_flags))),
def trace_unhandled(event_name, context, event_fields_dict):
try:
@@ -58,25 +61,26 @@ def trace_unhandled(event_name, context, event_fields_dict):
unhandled[event_name] = 1
def print_header(event_name, cpu, secs, nsecs, pid, comm):
- print "%-20s %5u %05u.%09u %8u %-20s " % \
- (event_name, cpu, secs, nsecs, pid, comm),
+print("%-20s %5u %05u.%09u %8u %-20s " %
+(event_name, cpu, secs, nsecs, pid, comm),
+end='')
# print trace fields not included in handler args
def print_uncommon(context):
-print "common_preempt_count=%d, common_flags=%s, common_lock_depth=%d, " \
-% (common_pc(context), trace_flag_str(common_flags(context)), \
- common_lock_depth(context))
+print("common_preempt_count=%d, common_flags=%s, common_lock_depth=%d, " %
+(common_pc(context), trace_flag_str(common_flags(context)),
+common_lock_depth(context))),
def print_unhandled():
keys = unhandled.keys()
if not keys:
return
-print "\nunhandled events:\n\n",
+print("\nunhandled events:\n")
-print "%-40s %10s\n" % ("event", "count"),
-print "%-40s %10s\n" % ("", \
- "---"),
+print("%-40s %10
[PATCH 10/15] perf script python: add Python3 support to sctop.py
Support both Python2 and Python3 in the sctop.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Tom Zanussi
---
tools/perf/scripts/python/sctop.py | 24
1 file changed, 16 insertions(+), 8 deletions(-)
diff --git a/tools/perf/scripts/python/sctop.py
b/tools/perf/scripts/python/sctop.py
index 61621b93affb..987ffae7c8ca 100644
--- a/tools/perf/scripts/python/sctop.py
+++ b/tools/perf/scripts/python/sctop.py
@@ -8,7 +8,14 @@
# will be refreshed every [interval] seconds. The default interval is
# 3 seconds.
-import os, sys, thread, time
+from __future__ import print_function
+
+import os, sys, time
+
+try:
+import thread
+except ImportError:
+import _thread as thread
sys.path.append(os.environ['PERF_EXEC_PATH'] + \
'/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
@@ -62,18 +69,19 @@ def print_syscall_totals(interval):
while 1:
clear_term()
if for_comm is not None:
- print "\nsyscall events for %s:\n\n" % (for_comm),
+ print("\nsyscall events for %s:\n" % (for_comm))
else:
- print "\nsyscall events:\n\n",
+ print("\nsyscall events:\n")
- print "%-40s %10s\n" % ("event", "count"),
- print "%-40s %10s\n" %
("", \
-"--"),
+ print("%-40s %10s" % ("event", "count"))
+ print("%-40s %10s" %
+("",
+"--"))
- for id, val in sorted(syscalls.iteritems(), key = lambda(k, v):
(v, k), \
+ for id, val in sorted(syscalls.items(), key = lambda kv:
(kv[1], kv[0]), \
reverse = True):
try:
- print "%-40s %10d\n" % (syscall_name(id), val),
+ print("%-40s %10d" % (syscall_name(id), val))
except TypeError:
pass
syscalls.clear()
--
2.20.1
[PATCH 04/15] perf script python: add Python3 support to failed-syscalls-by-pid.py
Support both Python2 and Python3 in the failed-syscalls-by-pid.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Tom Zanussi
---
.../scripts/python/failed-syscalls-by-pid.py | 21 ++-
1 file changed, 11 insertions(+), 10 deletions(-)
diff --git a/tools/perf/scripts/python/failed-syscalls-by-pid.py
b/tools/perf/scripts/python/failed-syscalls-by-pid.py
index cafeff3d74db..3648e8b986ec 100644
--- a/tools/perf/scripts/python/failed-syscalls-by-pid.py
+++ b/tools/perf/scripts/python/failed-syscalls-by-pid.py
@@ -5,6 +5,8 @@
# Displays system-wide failed system call totals, broken down by pid.
# If a [comm] arg is specified, only syscalls called by [comm] are displayed.
+from __future__ import print_function
+
import os
import sys
@@ -32,7 +34,7 @@ if len(sys.argv) > 1:
syscalls = autodict()
def trace_begin():
- print "Press control+C to stop and show the summary"
+ print("Press control+C to stop and show the summary")
def trace_end():
print_error_totals()
@@ -57,22 +59,21 @@ def syscalls__sys_exit(event_name, context, common_cpu,
def print_error_totals():
if for_comm is not None:
- print "\nsyscall errors for %s:\n\n" % (for_comm),
+ print("\nsyscall errors for %s:\n" % (for_comm))
else:
- print "\nsyscall errors:\n\n",
+ print("\nsyscall errors:\n")
-print "%-30s %10s\n" % ("comm [pid]", "count"),
-print "%-30s %10s\n" % ("--", \
- "--"),
+print("%-30s %10s" % ("comm [pid]", "count"))
+print("%-30s %10s" % ("--", "--"))
comm_keys = syscalls.keys()
for comm in comm_keys:
pid_keys = syscalls[comm].keys()
for pid in pid_keys:
- print "\n%s [%d]\n" % (comm, pid),
+ print("\n%s [%d]" % (comm, pid))
id_keys = syscalls[comm][pid].keys()
for id in id_keys:
- print " syscall: %-16s\n" % syscall_name(id),
+ print(" syscall: %-16s" % syscall_name(id))
ret_keys = syscalls[comm][pid][id].keys()
- for ret, val in
sorted(syscalls[comm][pid][id].iteritems(), key = lambda(k, v): (v, k),
reverse = True):
- print "err = %-20s %10d\n" %
(strerror(ret), val),
+ for ret, val in
sorted(syscalls[comm][pid][id].items(), key = lambda kv: (kv[1], kv[0]),
reverse = True):
+ print("err = %-20s %10d" %
(strerror(ret), val))
--
2.20.1
[PATCH 11/15] perf script python: add Python3 support to stackcollapse.py
Support both Python2 and Python3 in the stackcollapse.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Paolo Bonzini
---
tools/perf/scripts/python/stackcollapse.py | 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/tools/perf/scripts/python/stackcollapse.py
b/tools/perf/scripts/python/stackcollapse.py
index 1697b5e18c96..5e703efaddcc 100755
--- a/tools/perf/scripts/python/stackcollapse.py
+++ b/tools/perf/scripts/python/stackcollapse.py
@@ -19,6 +19,8 @@
# Written by Paolo Bonzini
# Based on Brendan Gregg's stackcollapse-perf.pl script.
+from __future__ import print_function
+
import os
import sys
from collections import defaultdict
@@ -120,7 +122,6 @@ def process_event(param_dict):
lines[stack_string] = lines[stack_string] + 1
def trace_end():
-list = lines.keys()
-list.sort()
+list = sorted(lines)
for stack in list:
-print "%s %d" % (stack, lines[stack])
+print("%s %d" % (stack, lines[stack]))
--
2.20.1
[PATCH 01/15] perf script python: add Python3 support to netdev-times.py
Support both Python2 and Python3 in the netdev-times.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Koki Sanagi
---
tools/perf/scripts/python/netdev-times.py | 82 ---
1 file changed, 42 insertions(+), 40 deletions(-)
diff --git a/tools/perf/scripts/python/netdev-times.py
b/tools/perf/scripts/python/netdev-times.py
index 9b2050f778f1..267bda49325d 100644
--- a/tools/perf/scripts/python/netdev-times.py
+++ b/tools/perf/scripts/python/netdev-times.py
@@ -8,6 +8,8 @@
# dev=: show only thing related to specified device
# debug: work with debug mode. It shows buffer status.
+from __future__ import print_function
+
import os
import sys
@@ -17,6 +19,7 @@ sys.path.append(os.environ['PERF_EXEC_PATH'] + \
from perf_trace_context import *
from Core import *
from Util import *
+from functools import cmp_to_key
all_event_list = []; # insert all tracepoint event related with this script
irq_dic = {}; # key is cpu and value is a list which stacks irqs
@@ -61,12 +64,12 @@ def diff_msec(src, dst):
def print_transmit(hunk):
if dev != 0 and hunk['dev'].find(dev) < 0:
return
- print "%7s %5d %6d.%06dsec %12.3fmsec %12.3fmsec" % \
+ print("%7s %5d %6d.%06dsec %12.3fmsec %12.3fmsec" %
(hunk['dev'], hunk['len'],
nsecs_secs(hunk['queue_t']),
nsecs_nsecs(hunk['queue_t'])/1000,
diff_msec(hunk['queue_t'], hunk['xmit_t']),
- diff_msec(hunk['xmit_t'], hunk['free_t']))
+ diff_msec(hunk['xmit_t'], hunk['free_t'])))
# Format for displaying rx packet processing
PF_IRQ_ENTRY= " irq_entry(+%.3fmsec irq=%d:%s)"
@@ -98,55 +101,55 @@ def print_receive(hunk):
if show_hunk == 0:
return
- print "%d.%06dsec cpu=%d" % \
- (nsecs_secs(base_t), nsecs_nsecs(base_t)/1000, cpu)
+ print("%d.%06dsec cpu=%d" %
+ (nsecs_secs(base_t), nsecs_nsecs(base_t)/1000, cpu))
for i in range(len(irq_list)):
- print PF_IRQ_ENTRY % \
+ print(PF_IRQ_ENTRY %
(diff_msec(base_t, irq_list[i]['irq_ent_t']),
- irq_list[i]['irq'], irq_list[i]['name'])
- print PF_JOINT
+ irq_list[i]['irq'], irq_list[i]['name']))
+ print(PF_JOINT)
irq_event_list = irq_list[i]['event_list']
for j in range(len(irq_event_list)):
irq_event = irq_event_list[j]
if irq_event['event'] == 'netif_rx':
- print PF_NET_RX % \
+ print(PF_NET_RX %
(diff_msec(base_t, irq_event['time']),
- irq_event['skbaddr'])
- print PF_JOINT
- print PF_SOFT_ENTRY % \
- diff_msec(base_t, hunk['sirq_ent_t'])
- print PF_JOINT
+ irq_event['skbaddr']))
+ print(PF_JOINT)
+ print(PF_SOFT_ENTRY %
+ diff_msec(base_t, hunk['sirq_ent_t']))
+ print(PF_JOINT)
event_list = hunk['event_list']
for i in range(len(event_list)):
event = event_list[i]
if event['event_name'] == 'napi_poll':
- print PF_NAPI_POLL % \
- (diff_msec(base_t, event['event_t']), event['dev'])
+ print(PF_NAPI_POLL %
+ (diff_msec(base_t, event['event_t']), event['dev']))
if i == len(event_list) - 1:
- print ""
+ print("")
else:
- print PF_JOINT
+ print(PF_JOINT)
else:
- print PF_NET_RECV % \
+ print(PF_NET_RECV %
(diff_msec(base_t, event['event_t']),
event['skbaddr'],
- event['len'])
+ event['len']))
if 'comm' in event.keys():
- print PF_WJOINT
- print PF_CPY_DGRAM % \
+ print(PF_WJOINT)
+ print(PF_CPY_DGRAM %
(diff_msec(base_t, event['comm_t']),
- event['pid'], event['comm'])
+ event['p
[PATCH 00/15] perf script python: add Python3 support
The following patches add Python3 support to the remainder of the scripts under scripts/python. Minimum supported Python version is now v2.6 I tested everything against v2.7 (no patch), v2.7 (patch) and v3.7 (patch). Seeteena also tested a subset (no SQL, no IPT) against v2.6 There are changes in order of output as key ordering is not consistent between v2 and v3 but format of each line should be unchanged.
[PATCH 05/15] perf script python: add Python3 support to futex-contention.py
Support both Python2 and Python3 in the futex-contention.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Arnaldo Carvalho de Melo
---
tools/perf/scripts/python/futex-contention.py | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/tools/perf/scripts/python/futex-contention.py
b/tools/perf/scripts/python/futex-contention.py
index 0f5cf437b602..665a9d5355b8 100644
--- a/tools/perf/scripts/python/futex-contention.py
+++ b/tools/perf/scripts/python/futex-contention.py
@@ -10,6 +10,8 @@
#
# Measures futex contention
+from __future__ import print_function
+
import os, sys
sys.path.append(os.environ['PERF_EXEC_PATH'] +
'/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
from Util import *
@@ -33,18 +35,18 @@ def syscalls__sys_enter_futex(event, ctxt, cpu, s, ns, tid,
comm, callchain,
def syscalls__sys_exit_futex(event, ctxt, cpu, s, ns, tid, comm, callchain,
nr, ret):
- if thread_blocktime.has_key(tid):
+ if tid in thread_blocktime:
elapsed = nsecs(s, ns) - thread_blocktime[tid]
add_stats(lock_waits, (tid, thread_thislock[tid]), elapsed)
del thread_blocktime[tid]
del thread_thislock[tid]
def trace_begin():
- print "Press control+C to stop and show the summary"
+ print("Press control+C to stop and show the summary")
def trace_end():
for (tid, lock) in lock_waits:
min, max, avg, count = lock_waits[tid, lock]
- print "%s[%d] lock %x contended %d times, %d avg ns" % \
- (process_names[tid], tid, lock, count, avg)
+ print("%s[%d] lock %x contended %d times, %d avg ns" %
+ (process_names[tid], tid, lock, count, avg))
--
2.20.1
[PATCH 07/15] perf script python: add Python3 support to mem-phys-addr.py
Support both Python2 and Python3 in the mem-phys-addr.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
---
tools/perf/scripts/python/mem-phys-addr.py | 24 +-
1 file changed, 14 insertions(+), 10 deletions(-)
diff --git a/tools/perf/scripts/python/mem-phys-addr.py
b/tools/perf/scripts/python/mem-phys-addr.py
index ebee2c5ae496..fb0bbcbfa0f0 100644
--- a/tools/perf/scripts/python/mem-phys-addr.py
+++ b/tools/perf/scripts/python/mem-phys-addr.py
@@ -4,6 +4,8 @@
# Copyright (c) 2018, Intel Corporation.
from __future__ import division
+from __future__ import print_function
+
import os
import sys
import struct
@@ -31,21 +33,23 @@ def parse_iomem():
for i, j in enumerate(f):
m = re.split('-|:',j,2)
if m[2].strip() == 'System RAM':
- system_ram.append(long(m[0], 16))
- system_ram.append(long(m[1], 16))
+ system_ram.append(int(m[0], 16))
+ system_ram.append(int(m[1], 16))
if m[2].strip() == 'Persistent Memory':
- pmem.append(long(m[0], 16))
- pmem.append(long(m[1], 16))
+ pmem.append(int(m[0], 16))
+ pmem.append(int(m[1], 16))
def print_memory_type():
- print "Event: %s" % (event_name)
- print "%-40s %10s %10s\n" % ("Memory type", "count", "percentage"),
- print "%-40s %10s %10s\n" %
("", \
+ print("Event: %s" % (event_name))
+ print("%-40s %10s %10s\n" % ("Memory type", "count", "percentage"),
end='')
+ print("%-40s %10s %10s\n" %
("",
"---", "---"),
+end='');
total = sum(load_mem_type_cnt.values())
for mem_type, count in sorted(load_mem_type_cnt.most_common(), \
- key = lambda(k, v): (v, k), reverse =
True):
- print "%-40s %10d %10.1f%%\n" % (mem_type, count, 100 * count
/ total),
+ key = lambda kv: (kv[1], kv[0]),
reverse = True):
+ print("%-40s %10d %10.1f%%\n" % (mem_type, count, 100 * count
/ total),
+end='')
def trace_begin():
parse_iomem()
@@ -80,7 +84,7 @@ def find_memory_type(phys_addr):
f.seek(0, 0)
for j in f:
m = re.split('-|:',j,2)
- if long(m[0], 16) <= phys_addr <= long(m[1], 16):
+ if int(m[0], 16) <= phys_addr <= int(m[1], 16):
return m[2]
return "N/A"
--
2.20.1
[PATCH 06/15] perf script python: add Python3 support to intel-pt-events.py
Support both Python2 and Python3 in the intel-pt-events.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
Fix space/tab inconsistency as python3 enforces consistency.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Adrian Hunter
---
tools/perf/scripts/python/intel-pt-events.py | 138 ++-
1 file changed, 70 insertions(+), 68 deletions(-)
diff --git a/tools/perf/scripts/python/intel-pt-events.py
b/tools/perf/scripts/python/intel-pt-events.py
index b19172d673af..aef54566af61 100644
--- a/tools/perf/scripts/python/intel-pt-events.py
+++ b/tools/perf/scripts/python/intel-pt-events.py
@@ -10,79 +10,81 @@
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
+from __future__ import print_function
+
import os
import sys
import struct
sys.path.append(os.environ['PERF_EXEC_PATH'] + \
- '/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
+'/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
# These perf imports are not used at present
#from perf_trace_context import *
#from Core import *
def trace_begin():
- print "Intel PT Power Events and PTWRITE"
+print("Intel PT Power Events and PTWRITE")
def trace_end():
- print "End"
+print("End")
def trace_unhandled(event_name, context, event_fields_dict):
- print ' '.join(['%s=%s'%(k,str(v))for k,v in
sorted(event_fields_dict.items())])
+print(' '.join(['%s=%s'%(k,str(v))for k,v in
sorted(event_fields_dict.items())]))
def print_ptwrite(raw_buf):
- data = struct.unpack_from("> 32) & 0x3
- print "hints: %#x extensions: %#x" % (hints, extensions),
+data = struct.unpack_from("> 32) & 0x3
+print("hints: %#x extensions: %#x" % (hints, extensions), end='')
def print_pwre(raw_buf):
- data = struct.unpack_from("> 7) & 1
- cstate = (payload >> 12) & 0xf
- subcstate = (payload >> 8) & 0xf
- print "hw: %u cstate: %u sub-cstate: %u" % (hw, cstate, subcstate),
+data = struct.unpack_from("> 7) & 1
+cstate = (payload >> 12) & 0xf
+subcstate = (payload >> 8) & 0xf
+print("hw: %u cstate: %u sub-cstate: %u" % (hw, cstate, subcstate),
end='')
def print_exstop(raw_buf):
- data = struct.unpack_from("> 4) & 0xf
- wake_reason = (payload >> 8) & 0xf
- print "deepest cstate: %u last cstate: %u wake reason: %#x" %
(deepest_cstate, last_cstate, wake_reason),
+data = struct.unpack_from("> 4) & 0xf
+wake_reason = (payload >> 8) & 0xf
+print("deepest cstate: %u last cstate: %u wake reason: %#x" %
(deepest_cstate, last_cstate, wake_reason), end='')
def print_common_start(comm, sample, name):
- ts = sample["time"]
- cpu = sample["cpu"]
- pid = sample["pid"]
- tid = sample["tid"]
- print "%16s %5u/%-5u [%03u] %9u.%09u %7s:" % (comm, pid, tid, cpu, ts /
10, ts %10, name),
+ts = sample["time"]
+cpu = sample["cpu"]
+pid = sample["pid"]
+tid = sample["tid"]
+print("%16s %5u/%-5u [%03u] %9u.%09u %7s:" % (comm, pid, tid, cpu, ts
/ 10, ts %10, name), end='')
def print_common_ip(sample, symbol, dso):
- ip = sample["ip"]
- print "%16x %s (%s)" % (ip, symbol, dso)
+ip = sample["ip"]
+print("%16x %s (%s)" % (ip, symbol, dso))
def process_event(param_dict):
event_attr = param_dict["attr"]
@@ -92,37 +94,37 @@ def process_event(param_dict):
name = param_dict["ev_name"]
# Symbol and dso info are not always resolved
-if (param_dict.has_key("dso")):
+if "dso" in param_dict:
dso = param_dict["dso"]
else:
dso = "[unknown]"
-if (param_dict.has_key("symbol")):
+if "symbol" in param_dict:
symbol = param_dict["symbol"]
else:
symbol = "[unknown]"
- if name == "ptwrite":
- print_common_start(comm, sample, name)
- print_ptwrite(raw_buf)
- print_common_ip(sample, symbol, dso)
- elif name == "cbr":
- print_common_start(comm, sample, name)
- print_cbr(raw_buf)
- print_c
[PATCH 13/15] perf script python: add Python3 support to syscall-counts.py
Support both Python2 and Python3 in the syscall-counts.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
---
tools/perf/scripts/python/syscall-counts.py | 18 ++
1 file changed, 10 insertions(+), 8 deletions(-)
diff --git a/tools/perf/scripts/python/syscall-counts.py
b/tools/perf/scripts/python/syscall-counts.py
index e66a7730aeb5..0ebd89cfd42c 100644
--- a/tools/perf/scripts/python/syscall-counts.py
+++ b/tools/perf/scripts/python/syscall-counts.py
@@ -5,6 +5,8 @@
# Displays system-wide system call totals, broken down by syscall.
# If a [comm] arg is specified, only syscalls called by [comm] are displayed.
+from __future__ import print_function
+
import os
import sys
@@ -28,7 +30,7 @@ if len(sys.argv) > 1:
syscalls = autodict()
def trace_begin():
- print "Press control+C to stop and show the summary"
+ print("Press control+C to stop and show the summary")
def trace_end():
print_syscall_totals()
@@ -51,14 +53,14 @@ def syscalls__sys_enter(event_name, context, common_cpu,
def print_syscall_totals():
if for_comm is not None:
- print "\nsyscall events for %s:\n\n" % (for_comm),
+ print("\nsyscall events for %s:\n" % (for_comm))
else:
- print "\nsyscall events:\n\n",
+ print("\nsyscall events:\n")
-print "%-40s %10s\n" % ("event", "count"),
-print "%-40s %10s\n" % ("", \
- "---"),
+print("%-40s %10s" % ("event", "count"))
+print("%-40s %10s" % ("",
+ "---"))
-for id, val in sorted(syscalls.iteritems(), key = lambda(k, v): (v, k), \
+for id, val in sorted(syscalls.items(), key = lambda kv: (kv[1], kv[0]), \
reverse = True):
- print "%-40s %10d\n" % (syscall_name(id), val),
+ print("%-40s %10d" % (syscall_name(id), val))
--
2.20.1
[PATCH 03/15] perf script python: add Python3 support to event_analyzing_sample.py
Support both Python2 and Python3 in the event_analyzing_sample.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
---
.../scripts/python/event_analyzing_sample.py | 48 ++-
1 file changed, 25 insertions(+), 23 deletions(-)
diff --git a/tools/perf/scripts/python/event_analyzing_sample.py
b/tools/perf/scripts/python/event_analyzing_sample.py
index 4e843b9864ec..f4c4c7963451 100644
--- a/tools/perf/scripts/python/event_analyzing_sample.py
+++ b/tools/perf/scripts/python/event_analyzing_sample.py
@@ -15,6 +15,8 @@
# for a x86 HW PMU event: PEBS with load latency data.
#
+from __future__ import print_function
+
import os
import sys
import math
@@ -37,7 +39,7 @@ con = sqlite3.connect("/dev/shm/perf.db")
con.isolation_level = None
def trace_begin():
- print "In trace_begin:\n"
+print("In trace_begin:\n")
#
# Will create several tables at the start, pebs_ll is for PEBS data
with
@@ -76,12 +78,12 @@ def process_event(param_dict):
name = param_dict["ev_name"]
# Symbol and dso info are not always resolved
-if (param_dict.has_key("dso")):
+if ("dso" in param_dict):
dso = param_dict["dso"]
else:
dso = "Unknown_dso"
-if (param_dict.has_key("symbol")):
+if ("symbol" in param_dict):
symbol = param_dict["symbol"]
else:
symbol = "Unknown_symbol"
@@ -102,7 +104,7 @@ def insert_db(event):
event.ip, event.status, event.dse, event.dla,
event.lat))
def trace_end():
- print "In trace_end:\n"
+print("In trace_end:\n")
# We show the basic info for the 2 type of event classes
show_general_events()
show_pebs_ll()
@@ -123,29 +125,29 @@ def show_general_events():
# Check the total record number in the table
count = con.execute("select count(*) from gen_events")
for t in count:
-print "There is %d records in gen_events table" % t[0]
+print("There is %d records in gen_events table" % t[0])
if t[0] == 0:
return
-print "Statistics about the general events grouped by
thread/symbol/dso: \n"
+print("Statistics about the general events grouped by
thread/symbol/dso: \n")
# Group by thread
commq = con.execute("select comm, count(comm) from gen_events group by
comm order by -count(comm)")
-print "\n%16s %8s %16s\n%s" % ("comm", "number", "histogram", "="*42)
+print("\n%16s %8s %16s\n%s" % ("comm", "number", "histogram", "="*42))
for row in commq:
- print "%16s %8d %s" % (row[0], row[1], num2sym(row[1]))
+ print("%16s %8d %s" % (row[0], row[1], num2sym(row[1])))
# Group by symbol
-print "\n%32s %8s %16s\n%s" % ("symbol", "number", "histogram", "="*58)
+print("\n%32s %8s %16s\n%s" % ("symbol", "number", "histogram",
"="*58))
symbolq = con.execute("select symbol, count(symbol) from gen_events
group by symbol order by -count(symbol)")
for row in symbolq:
- print "%32s %8d %s" % (row[0], row[1], num2sym(row[1]))
+ print("%32s %8d %s" % (row[0], row[1], num2sym(row[1])))
# Group by dso
-print "\n%40s %8s %16s\n%s" % ("dso", "number", "histogram", "="*74)
+print("\n%40s %8s %16s\n%s" % ("dso", "number", "histogram", "="*74))
dsoq = con.execute("select dso, count(dso) from gen_events group by
dso order by -count(dso)")
for row in dsoq:
- print "%40s %8d %s" % (row[0], row[1], num2sym(row[1]))
+ print("%40s %8d %s" % (row[0], row[1], num2sym(row[1])))
#
# This function just shows the basic info, and we could do more with the
@@ -156,35 +158,35 @@ def show_pebs_ll():
count = con.execute("select count(*) from pebs_ll")
for t in count:
-print "There is %d records in pebs_ll table" % t[0]
+print("There is
[PATCH 12/15] perf script python: add Python3 support to stat-cpi.py
Support both Python2 and Python3 in the stat-cpi.py script
There may be differences in the ordering of output lines due to
differences in dictionary ordering etc. However the format within lines
should be unchanged.
The use of 'from __future__' implies the minimum supported Python2 version
is now v2.6
Signed-off-by: Tony Jones
Signed-off-by: Seeteena Thoufeek
Cc: Jiri Olsa
---
tools/perf/scripts/python/stat-cpi.py | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/tools/perf/scripts/python/stat-cpi.py
b/tools/perf/scripts/python/stat-cpi.py
index a81ad8835a74..01fa933ff3cf 100644
--- a/tools/perf/scripts/python/stat-cpi.py
+++ b/tools/perf/scripts/python/stat-cpi.py
@@ -1,5 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
+from __future__ import print_function
+
data= {}
times = []
threads = []
@@ -19,8 +21,8 @@ def store_key(time, cpu, thread):
threads.append(thread)
def store(time, event, cpu, thread, val, ena, run):
-#print "event %s cpu %d, thread %d, time %d, val %d, ena %d, run %d" % \
-# (event, cpu, thread, time, val, ena, run)
+#print("event %s cpu %d, thread %d, time %d, val %d, ena %d, run %d" %
+# (event, cpu, thread, time, val, ena, run))
store_key(time, cpu, thread)
key = get_key(time, event, cpu, thread)
@@ -58,7 +60,7 @@ def stat__interval(time):
if ins != 0:
cpi = cyc/float(ins)
-print "%15f: cpu %d, thread %d -> cpi %f (%d/%d)" %
(time/(float(10)), cpu, thread, cpi, cyc, ins)
+print("%15f: cpu %d, thread %d -> cpi %f (%d/%d)" %
(time/(float(10)), cpu, thread, cpi, cyc, ins))
def trace_end():
pass
@@ -74,4 +76,4 @@ def trace_end():
#if ins != 0:
#cpi = cyc/float(ins)
#
-#print "time %.9f, cpu %d, thread %d -> cpi %f" %
(time/(float(10)), cpu, thread, cpi)
+#print("time %.9f, cpu %d, thread %d -> cpi %f" %
(time/(float(10)), cpu, thread, cpi))
--
2.20.1
[tip:perf/urgent] perf script python: Add Python3 support to tests/attr.py
Commit-ID: 8f2f350cbdb2c2fbff654cb778139144b48a59ba
Gitweb: https://git.kernel.org/tip/8f2f350cbdb2c2fbff654cb778139144b48a59ba
Author: Tony Jones
AuthorDate: Wed, 23 Jan 2019 16:52:29 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Tue, 5 Feb 2019 10:31:08 -0300
perf script python: Add Python3 support to tests/attr.py
Support both Python 2 and Python 3 in tests/attr.py
The use of "except as" syntax implies the minimum supported Python2 version is
now v2.6
Committer testing:
$ make -C tools/perf PYTHON3=python install-bin
Before:
# perf test attr
16: Setup struct perf_event_attr : FAILED!
48: Synthesize attr update: Ok
[root@quaco ~]# perf test -v attr
16: Setup struct perf_event_attr :
--- start ---
test child forked, pid 3121
File "/home/acme/libexec/perf-core/tests/attr.py", line 324
except Unsup, obj:
^
SyntaxError: invalid syntax
test child finished with -1
end
Setup struct perf_event_attr: FAILED!
48: Synthesize attr update:
--- start ---
test child forked, pid 3124
test child finished with 0
end
Synthesize attr update: Ok
#
After:
# perf test attr
16: Setup struct perf_event_attr : Ok
48: Synthesize attr update: Ok
#
Signed-off-by: Tony Jones
Acked-by: Jiri Olsa
Tested-by: Arnaldo Carvalho de Melo
Cc: Jonathan Corbet
Cc: Ravi Bangoria
Cc: Seeteena Thoufeek
Link: http://lkml.kernel.org/r/20190124005229.16146-7-to...@suse.de
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/tests/attr.py | 32 +++-
1 file changed, 19 insertions(+), 13 deletions(-)
diff --git a/tools/perf/tests/attr.py b/tools/perf/tests/attr.py
index 44090a9a19f3..e952127e4fb0 100644
--- a/tools/perf/tests/attr.py
+++ b/tools/perf/tests/attr.py
@@ -1,6 +1,8 @@
#! /usr/bin/python
# SPDX-License-Identifier: GPL-2.0
+from __future__ import print_function
+
import os
import sys
import glob
@@ -8,7 +10,11 @@ import optparse
import tempfile
import logging
import shutil
-import ConfigParser
+
+try:
+import configparser
+except ImportError:
+import ConfigParser as configparser
def data_equal(a, b):
# Allow multiple values in assignment separated by '|'
@@ -100,20 +106,20 @@ class Event(dict):
def equal(self, other):
for t in Event.terms:
log.debug(" [%s] %s %s" % (t, self[t], other[t]));
-if not self.has_key(t) or not other.has_key(t):
+if t not in self or t not in other:
return False
if not data_equal(self[t], other[t]):
return False
return True
def optional(self):
-if self.has_key('optional') and self['optional'] == '1':
+if 'optional' in self and self['optional'] == '1':
return True
return False
def diff(self, other):
for t in Event.terms:
-if not self.has_key(t) or not other.has_key(t):
+if t not in self or t not in other:
continue
if not data_equal(self[t], other[t]):
log.warning("expected %s=%s, got %s" % (t, self[t], other[t]))
@@ -134,7 +140,7 @@ class Event(dict):
# - expected values assignments
class Test(object):
def __init__(self, path, options):
-parser = ConfigParser.SafeConfigParser()
+parser = configparser.SafeConfigParser()
parser.read(path)
log.warning("running '%s'" % path)
@@ -193,7 +199,7 @@ class Test(object):
return True
def load_events(self, path, events):
-parser_event = ConfigParser.SafeConfigParser()
+parser_event = configparser.SafeConfigParser()
parser_event.read(path)
# The event record section header contains 'event' word,
@@ -207,7 +213,7 @@ class Test(object):
# Read parent event if there's any
if (':' in section):
base = section[section.index(':') + 1:]
-parser_base = ConfigParser.SafeConfigParser()
+parser_base = configparser.SafeConfigParser()
parser_base.read(self.test_dir + '/' + base)
base_items = parser_base.items('event')
@@ -322,9 +328,9 @@ def run_tests(options):
for f in glob.glob(options.test_dir + '/' + options.test):
try:
Test(f, options).run()
-except Unsup, obj:
+except Unsup as obj:
log.warning("unsupp %s" % obj.getMsg())
-except Notest, obj:
+except Notest as obj:
log.warning("skipped %s" % obj.getMsg())
def setup_log(verbose):
@@ -363,7 +369,7 @@ def main():
parser.add_option("-p", "--perf",
[tip:perf/core] perf script python: Add Python3 support to tests/attr.py
Commit-ID: 35ea7e4bbb89ecd32057f5f6a2a8feb0d7224e51
Gitweb: https://git.kernel.org/tip/35ea7e4bbb89ecd32057f5f6a2a8feb0d7224e51
Author: Tony Jones
AuthorDate: Wed, 23 Jan 2019 16:52:29 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Fri, 25 Jan 2019 15:12:10 +0100
perf script python: Add Python3 support to tests/attr.py
Support both Python 2 and Python 3 in tests/attr.py
The use of "except as" syntax implies the minimum supported Python2 version is
now v2.6
Committer testing:
$ make -C tools/perf PYTHON3=python install-bin
Before:
# perf test attr
16: Setup struct perf_event_attr : FAILED!
48: Synthesize attr update: Ok
[root@quaco ~]# perf test -v attr
16: Setup struct perf_event_attr :
--- start ---
test child forked, pid 3121
File "/home/acme/libexec/perf-core/tests/attr.py", line 324
except Unsup, obj:
^
SyntaxError: invalid syntax
test child finished with -1
end
Setup struct perf_event_attr: FAILED!
48: Synthesize attr update:
--- start ---
test child forked, pid 3124
test child finished with 0
end
Synthesize attr update: Ok
#
After:
# perf test attr
16: Setup struct perf_event_attr : Ok
48: Synthesize attr update: Ok
#
Signed-off-by: Tony Jones
Acked-by: Jiri Olsa
Tested-by: Arnaldo Carvalho de Melo
Cc: Jonathan Corbet
Cc: Ravi Bangoria
Cc: Seeteena Thoufeek
Link: http://lkml.kernel.org/r/20190124005229.16146-7-to...@suse.de
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/tests/attr.py | 32 +++-
1 file changed, 19 insertions(+), 13 deletions(-)
diff --git a/tools/perf/tests/attr.py b/tools/perf/tests/attr.py
index 3e07eee33b10..cb39ac46bc73 100644
--- a/tools/perf/tests/attr.py
+++ b/tools/perf/tests/attr.py
@@ -1,5 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
+from __future__ import print_function
+
import os
import sys
import glob
@@ -7,7 +9,11 @@ import optparse
import tempfile
import logging
import shutil
-import ConfigParser
+
+try:
+import configparser
+except ImportError:
+import ConfigParser as configparser
def data_equal(a, b):
# Allow multiple values in assignment separated by '|'
@@ -99,20 +105,20 @@ class Event(dict):
def equal(self, other):
for t in Event.terms:
log.debug(" [%s] %s %s" % (t, self[t], other[t]));
-if not self.has_key(t) or not other.has_key(t):
+if t not in self or t not in other:
return False
if not data_equal(self[t], other[t]):
return False
return True
def optional(self):
-if self.has_key('optional') and self['optional'] == '1':
+if 'optional' in self and self['optional'] == '1':
return True
return False
def diff(self, other):
for t in Event.terms:
-if not self.has_key(t) or not other.has_key(t):
+if t not in self or t not in other:
continue
if not data_equal(self[t], other[t]):
log.warning("expected %s=%s, got %s" % (t, self[t], other[t]))
@@ -133,7 +139,7 @@ class Event(dict):
# - expected values assignments
class Test(object):
def __init__(self, path, options):
-parser = ConfigParser.SafeConfigParser()
+parser = configparser.SafeConfigParser()
parser.read(path)
log.warning("running '%s'" % path)
@@ -192,7 +198,7 @@ class Test(object):
return True
def load_events(self, path, events):
-parser_event = ConfigParser.SafeConfigParser()
+parser_event = configparser.SafeConfigParser()
parser_event.read(path)
# The event record section header contains 'event' word,
@@ -206,7 +212,7 @@ class Test(object):
# Read parent event if there's any
if (':' in section):
base = section[section.index(':') + 1:]
-parser_base = ConfigParser.SafeConfigParser()
+parser_base = configparser.SafeConfigParser()
parser_base.read(self.test_dir + '/' + base)
base_items = parser_base.items('event')
@@ -321,9 +327,9 @@ def run_tests(options):
for f in glob.glob(options.test_dir + '/' + options.test):
try:
Test(f, options).run()
-except Unsup, obj:
+except Unsup as obj:
log.warning("unsupp %s" % obj.getMsg())
-except Notest, obj:
+except Notest as obj:
log.warning("skipped %s" % obj.getMsg())
def setup_log(verbose):
@@ -362,7 +368,7 @@ def main():
parser.add_option("-p", "--perf",
[tip:perf/core] perf script python: Remove explicit shebang from Python scripts
Commit-ID: a38352de4495a6a4662609a560b2db4b03d6b352
Gitweb: https://git.kernel.org/tip/a38352de4495a6a4662609a560b2db4b03d6b352
Author: Tony Jones
AuthorDate: Wed, 23 Jan 2019 16:52:28 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Fri, 25 Jan 2019 15:12:10 +0100
perf script python: Remove explicit shebang from Python scripts
The scripts in scripts/python are intended to be run from 'perf script'
and the Python version used is dictated by how perf was built (PYTHON=).
Also most distros follow pep-0394 which recommends that /usr/bin/python
refer to Python2 and so may not exist on the system (if PYTHON=python3).
- Remove the explicit shebang
- Install the scripts as mode 644
Signed-off-by: Tony Jones
Acked-by: Jiri Olsa
Cc: Adrian Hunter
Cc: Frederic Weisbecker
Cc: Jonathan Corbet
Cc: Ravi Bangoria
Cc: Seeteena Thoufeek
Link: http://lkml.kernel.org/r/20190124005229.16146-6-to...@suse.de
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/Makefile.perf | 4 ++--
tools/perf/scripts/python/exported-sql-viewer.py | 1 -
tools/perf/scripts/python/sched-migration.py | 2 --
tools/perf/scripts/python/stat-cpi.py| 1 -
4 files changed, 2 insertions(+), 6 deletions(-)
diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index 0ee6795d82cc..09df1c8a4ec9 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -863,8 +863,8 @@ ifndef NO_LIBPYTHON
$(call QUIET_INSTALL, python-scripts) \
$(INSTALL) -d -m 755
'$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python/Perf-Trace-Util/lib/Perf/Trace';
\
$(INSTALL) -d -m 755
'$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python/bin'; \
- $(INSTALL) scripts/python/Perf-Trace-Util/lib/Perf/Trace/* -t
'$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python/Perf-Trace-Util/lib/Perf/Trace';
\
- $(INSTALL) scripts/python/*.py -t
'$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python'; \
+ $(INSTALL) scripts/python/Perf-Trace-Util/lib/Perf/Trace/* -m
644 -t
'$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python/Perf-Trace-Util/lib/Perf/Trace';
\
+ $(INSTALL) scripts/python/*.py -m 644 -t
'$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python'; \
$(INSTALL) scripts/python/bin/* -t
'$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python/bin'
endif
$(call QUIET_INSTALL, perf_completion-script) \
diff --git a/tools/perf/scripts/python/exported-sql-viewer.py
b/tools/perf/scripts/python/exported-sql-viewer.py
index f278ce5ebab7..c3091401df91 100755
--- a/tools/perf/scripts/python/exported-sql-viewer.py
+++ b/tools/perf/scripts/python/exported-sql-viewer.py
@@ -1,4 +1,3 @@
-#!/usr/bin/python2
# SPDX-License-Identifier: GPL-2.0
# exported-sql-viewer.py: view data from sql database
# Copyright (c) 2014-2018, Intel Corporation.
diff --git a/tools/perf/scripts/python/sched-migration.py
b/tools/perf/scripts/python/sched-migration.py
index 3473e7f66081..3984bf51f3c5 100644
--- a/tools/perf/scripts/python/sched-migration.py
+++ b/tools/perf/scripts/python/sched-migration.py
@@ -1,5 +1,3 @@
-#!/usr/bin/python
-#
# Cpu task migration overview toy
#
# Copyright (C) 2010 Frederic Weisbecker
diff --git a/tools/perf/scripts/python/stat-cpi.py
b/tools/perf/scripts/python/stat-cpi.py
index 8410672efb8b..a81ad8835a74 100644
--- a/tools/perf/scripts/python/stat-cpi.py
+++ b/tools/perf/scripts/python/stat-cpi.py
@@ -1,4 +1,3 @@
-#!/usr/bin/env python
# SPDX-License-Identifier: GPL-2.0
data= {}
[tip:perf/core] perf script python: Remove explicit shebang from tests/attr.c
Commit-ID: d72eadbc1d2866fc047edd4535ffb0298fe240be Gitweb: https://git.kernel.org/tip/d72eadbc1d2866fc047edd4535ffb0298fe240be Author: Tony Jones AuthorDate: Wed, 23 Jan 2019 16:52:27 -0800 Committer: Arnaldo Carvalho de Melo CommitDate: Fri, 25 Jan 2019 15:12:10 +0100 perf script python: Remove explicit shebang from tests/attr.c tests/attr.c invokes attr.py via an explicit invocation of Python ($PYTHON) so there is therefore no need for an explicit shebang. Also most distros follow pep-0394 which recommends that /usr/bin/python refer only to v2 and so may not exist on the system (if PYTHON=python3). Signed-off-by: Tony Jones Acked-by: Jiri Olsa Cc: Jonathan Corbet Cc: Ravi Bangoria Cc: Seeteena Thoufeek Link: http://lkml.kernel.org/r/20190124005229.16146-5-to...@suse.de Signed-off-by: Arnaldo Carvalho de Melo --- tools/perf/tests/attr.py | 1 - 1 file changed, 1 deletion(-) diff --git a/tools/perf/tests/attr.py b/tools/perf/tests/attr.py index 44090a9a19f3..3e07eee33b10 100644 --- a/tools/perf/tests/attr.py +++ b/tools/perf/tests/attr.py @@ -1,4 +1,3 @@ -#! /usr/bin/python # SPDX-License-Identifier: GPL-2.0 import os
[tip:perf/core] perf script python: Remove explicit shebang from setup.py
Commit-ID: 099b79ca25c507ecbb25fb04f434e10415b68de0 Gitweb: https://git.kernel.org/tip/099b79ca25c507ecbb25fb04f434e10415b68de0 Author: Tony Jones AuthorDate: Wed, 23 Jan 2019 16:52:26 -0800 Committer: Arnaldo Carvalho de Melo CommitDate: Fri, 25 Jan 2019 15:12:10 +0100 perf script python: Remove explicit shebang from setup.py Makefile.perf invokes setup.py via an explicit invocation of python (PYTHON_WORD) so there is therefore no need for an explicit shebang. Also most distros follow pep-0394 which recommends that /usr/bin/python refer only to v2 and so may not exist on the system (if PYTHON=python3). Signed-off-by: Tony Jones Acked-by: Jiri Olsa Cc: Jonathan Corbet Cc: Ravi Bangoria Cc: Seeteena Thoufeek Link: http://lkml.kernel.org/r/20190124005229.16146-4-to...@suse.de Signed-off-by: Arnaldo Carvalho de Melo --- tools/perf/util/setup.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/tools/perf/util/setup.py b/tools/perf/util/setup.py index 64d1f36dee99..d3ffc18424b5 100644 --- a/tools/perf/util/setup.py +++ b/tools/perf/util/setup.py @@ -1,5 +1,3 @@ -#!/usr/bin/python - from os import getenv from subprocess import Popen, PIPE from re import sub
[tip:perf/core] perf script python: Use PyBytes for attr in trace-event-python
Commit-ID: 72e0b15cb24a497d7d0d4707cf51ff40c185ae8c
Gitweb: https://git.kernel.org/tip/72e0b15cb24a497d7d0d4707cf51ff40c185ae8c
Author: Tony Jones
AuthorDate: Wed, 23 Jan 2019 16:52:25 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Fri, 25 Jan 2019 15:12:10 +0100
perf script python: Use PyBytes for attr in trace-event-python
With Python3. PyUnicode_FromStringAndSize is unsafe to call on attr and will
return NULL. Use _PyBytes_FromStringAndSize (as with raw_buf).
Below is the observed behavior without the fix. Note it is first necessary
to apply the prior fix (Add trace_context extension module to sys,modules):
# ldd /usr/bin/perf | grep -i python
libpython3.6m.so.1.0 => /usr/lib64/libpython3.6m.so.1.0
(0x7f8e1dfb2000)
# perf record -e raw_syscalls:sys_enter /bin/false
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.018 MB perf.data (21 samples) ]
# perf script -g python | cat
generated Python script: perf-script.py
# perf script -s ./perf-script.py
in trace_begin
Segmentation fault (core dumped)
Signed-off-by: Tony Jones
Acked-by: Jiri Olsa
Tested-by: Arnaldo Carvalho de Melo
Cc: Jaroslav Škarvada
Cc: Jonathan Corbet
Cc: Ravi Bangoria
Cc: Seeteena Thoufeek
Fixes: 66dfdff03d19 ("perf tools: Add Python 3 support")
Link: http://lkml.kernel.org/r/20190124005229.16146-3-to...@suse.de
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/util/scripting-engines/trace-event-python.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c
b/tools/perf/util/scripting-engines/trace-event-python.c
index 315905c748fa..7059d1be2d09 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -733,8 +733,7 @@ static PyObject *get_perf_sample_dict(struct perf_sample
*sample,
Py_FatalError("couldn't create Python dictionary");
pydict_set_item_string_decref(dict, "ev_name",
_PyUnicode_FromString(perf_evsel__name(evsel)));
- pydict_set_item_string_decref(dict, "attr",
_PyUnicode_FromStringAndSize(
- (const char *)>attr, sizeof(evsel->attr)));
+ pydict_set_item_string_decref(dict, "attr",
_PyBytes_FromStringAndSize((const char *)>attr, sizeof(evsel->attr)));
pydict_set_item_string_decref(dict_sample, "pid",
_PyLong_FromLong(sample->pid));
[tip:perf/core] perf script python: Add trace_context extension module to sys.modules
Commit-ID: cc437642255224e4140fed1f3e3156fc8ad91903
Gitweb: https://git.kernel.org/tip/cc437642255224e4140fed1f3e3156fc8ad91903
Author: Tony Jones
AuthorDate: Wed, 23 Jan 2019 16:52:24 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Fri, 25 Jan 2019 15:12:10 +0100
perf script python: Add trace_context extension module to sys.modules
In Python3, the result of PyModule_Create (called from
scripts/python/Perf-Trace-Util/Context.c) is not automatically added to
sys.modules. See: https://bugs.python.org/issue4592
Below is the observed behavior without the fix:
# ldd /usr/bin/perf | grep -i python
libpython3.6m.so.1.0 => /usr/lib64/libpython3.6m.so.1.0
(0x7f8e1dfb2000)
# perf record /bin/false
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.015 MB perf.data (17 samples) ]
# perf script -g python | cat
generated Python script: perf-script.py
# perf script -s ./perf-script.py
Traceback (most recent call last):
File "./perf-script.py", line 18, in
from perf_trace_context import *
ModuleNotFoundError: No module named 'perf_trace_context'
Error running python script ./perf-script.py
#
Committer notes:
To build with python3 use:
$ make -C tools/perf PYTHON=python3
Use a non-const variable to pass the 'name' arg to
PyImport_AppendInittab(), as python2.6 has that as 'char *', which ends
up trowing this in some environments:
CC /tmp/build/perf/util/parse-branch-options.o
util/scripting-engines/trace-event-python.c: In function
'python_start_script':
util/scripting-engines/trace-event-python.c:1520:2: error: passing argument 1
of 'PyImport_AppendInittab' discards 'const' qualifier from pointer target type
[-Werror]
PyImport_AppendInittab("perf_trace_context", initfunc);
^
In file included from /usr/include/python2.6/Python.h:130:0,
from util/scripting-engines/trace-event-python.c:22:
/usr/include/python2.6/import.h:54:17: note: expected 'char *' but argument
is of type 'const char *'
PyAPI_FUNC(int) PyImport_AppendInittab(char *name, void (*initfunc)(void));
^
cc1: all warnings being treated as errors
Signed-off-by: Tony Jones
Acked-by: Jiri Olsa
Tested-by: Arnaldo Carvalho de Melo
Cc: Jaroslav Škarvada
Cc: Jonathan Corbet
Cc: Ravi Bangoria
Cc: Seeteena Thoufeek
Fixes: 66dfdff03d19 ("perf tools: Add Python 3 support")
Link: http://lkml.kernel.org/r/20190124005229.16146-2-to...@suse.de
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/util/scripting-engines/trace-event-python.c | 12 +---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c
b/tools/perf/util/scripting-engines/trace-event-python.c
index 87ef16a1b17e..315905c748fa 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -1494,34 +1494,40 @@ static void _free_command_line(wchar_t **command_line,
int num)
static int python_start_script(const char *script, int argc, const char **argv)
{
struct tables *tables = _global;
+ PyMODINIT_FUNC (*initfunc)(void);
#if PY_MAJOR_VERSION < 3
const char **command_line;
#else
wchar_t **command_line;
#endif
- char buf[PATH_MAX];
+ /*
+* Use a non-const name variable to cope with python 2.6's
+* PyImport_AppendInittab prototype
+*/
+ char buf[PATH_MAX], name[19] = "perf_trace_context";
int i, err = 0;
FILE *fp;
#if PY_MAJOR_VERSION < 3
+ initfunc = initperf_trace_context;
command_line = malloc((argc + 1) * sizeof(const char *));
command_line[0] = script;
for (i = 1; i < argc + 1; i++)
command_line[i] = argv[i - 1];
#else
+ initfunc = PyInit_perf_trace_context;
command_line = malloc((argc + 1) * sizeof(wchar_t *));
command_line[0] = Py_DecodeLocale(script, NULL);
for (i = 1; i < argc + 1; i++)
command_line[i] = Py_DecodeLocale(argv[i - 1], NULL);
#endif
+ PyImport_AppendInittab(name, initfunc);
Py_Initialize();
#if PY_MAJOR_VERSION < 3
- initperf_trace_context();
PySys_SetArgv(argc + 1, (char **)command_line);
#else
- PyInit_perf_trace_context();
PySys_SetArgv(argc + 1, command_line);
#endif
Re: [PATCH 0/6] Fix issues with Python3 scripting
On 1/25/19 5:57 AM, Arnaldo Carvalho de Melo wrote:
> Em Fri, Jan 25, 2019 at 01:31:19PM +0100, Arnaldo Carvalho de Melo escreveu:
>> Em Wed, Jan 23, 2019 at 04:52:23PM -0800, Tony Jones escreveu:
>>> Seeteena posted, earlier this week, some patches to add Python3 support
>>> to scripts/python/*.py. Unfortunately there were some issues with these
>>> patches (such as: https://lkml.org/lkml/2019/1/17/351)
>>>
>>> Since I already had a tested set of patches in openSUSE:Factory and
>>> SLE15-SP1 and was about to submit them, Seeteena and I that agreed I
>>> should post my patches not involving scripts/python/*.py and Seeteena
>>> will later resubmit the patches for scripts/python/*.py incorporating
>>> my review feedback under a joint signed-off-by.
>>>
>>> It should be noted that the use of "from __future__ import print_function"
>>> (see: https://lkml.org/lkml/2019/1/16/641) and "except as" (see change to:
>>> tests/attr.py) implies Python2 >= 2.6 as the necessary support has not
>>> been backported to prior versions. I am not sure if it's worth detecting
>>> <2.6 at build time or whether it's sufficiently old as to be a non-issue?
>>>
>>> The shebang changes were driven mostly by our build process as it scans
>>> all files within an rpm and the shebangs would result in a rpm requires
>>> on the python2 binary when BuildRequires was python3-devel. I think they
>>> make sense to apply upstream but understand totally if it's prefered we
>>> keep them local.
>>>
>>> These changes have been tested with PYTHON=python2 (v2.7) and
>>> PYTHON=python3 (v3.6) on latest openSUSE Tumbleweed. I did notice that
>>> test #18 "'import perf' in python" is failing on my system without these
>>> changes. I'll look at it further but didn't want to hold up Seeteena's
>>> resubmit.
>>
>> So it fails on AmazonLinux 1, that has python 2.6, please check if this
>> is something we can workaround, if its difficult, I'll just use
>> NO_PYTHON=1 there to disable it.
Sorry about this. I'll be sure to test more broadly next time.
>>
>> CC /tmp/build/perf/util/parse-branch-options.o
>> util/scripting-engines/trace-event-python.c: In function
>> 'python_start_script':
>> util/scripting-engines/trace-event-python.c:1520:2: error: passing argument
>> 1 of 'PyImport_AppendInittab' discards 'const' qualifier from pointer target
>> type [-Werror]
>> PyImport_AppendInittab("perf_trace_context", initfunc);
>> ^
>> In file included from /usr/include/python2.6/Python.h:130:0,
>> from util/scripting-engines/trace-event-python.c:22:
>> /usr/include/python2.6/import.h:54:17: note: expected 'char *' but argument
>> is of type 'const char *'
>> PyAPI_FUNC(int) PyImport_AppendInittab(char *name, void (*initfunc)(void));
>> ^
>> cc1: all warnings being treated as errors
>> mv: cannot stat
>> '/tmp/build/perf/util/scripting-engines/.trace-event-python.o.tmp': No such
>> file or directory
>> make[5]: *** [/tmp/build/perf/util/scripting-engines/trace-event-python.o]
>> Error 1
>
> I did a quick hack to init an auto variable with that const string and
> then pass it, is passing everything so far:
I see that you already amended in cc4376422552
Thanks!
[PATCH 4/6] perf script python: remove explicit shebang from tests/attr.c
tests/attr.c invokes attr.py via an explicit invocation of Python ($PYTHON) so there is therefore no need for an explicit shebang. Also most distros follow pep-0394 which recommends that /usr/bin/python refer only to v2 and so may not exist on the system (if PYTHON=python3). Signed-off-By: Tony Jones Cc: Jiri Olsa --- tools/perf/tests/attr.py | 1 - 1 file changed, 1 deletion(-) diff --git a/tools/perf/tests/attr.py b/tools/perf/tests/attr.py index 44090a9a19f3..3e07eee33b10 100644 --- a/tools/perf/tests/attr.py +++ b/tools/perf/tests/attr.py @@ -1,4 +1,3 @@ -#! /usr/bin/python # SPDX-License-Identifier: GPL-2.0 import os -- 2.20.1
[PATCH 6/6] perf script python: add Python3 support to tests/attr.py
Support both Python 2 and Python 3 in tests/attr.py
The use of "except as" syntax implies the minimum supported Python2 version is
now v2.6
Signed-off-By: Tony Jones
Cc: Jiri Olsa
---
tools/perf/tests/attr.py | 32 +++-
1 file changed, 19 insertions(+), 13 deletions(-)
diff --git a/tools/perf/tests/attr.py b/tools/perf/tests/attr.py
index 3e07eee33b10..cb39ac46bc73 100644
--- a/tools/perf/tests/attr.py
+++ b/tools/perf/tests/attr.py
@@ -1,5 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
+from __future__ import print_function
+
import os
import sys
import glob
@@ -7,7 +9,11 @@ import optparse
import tempfile
import logging
import shutil
-import ConfigParser
+
+try:
+import configparser
+except ImportError:
+import ConfigParser as configparser
def data_equal(a, b):
# Allow multiple values in assignment separated by '|'
@@ -99,20 +105,20 @@ class Event(dict):
def equal(self, other):
for t in Event.terms:
log.debug(" [%s] %s %s" % (t, self[t], other[t]));
-if not self.has_key(t) or not other.has_key(t):
+if t not in self or t not in other:
return False
if not data_equal(self[t], other[t]):
return False
return True
def optional(self):
-if self.has_key('optional') and self['optional'] == '1':
+if 'optional' in self and self['optional'] == '1':
return True
return False
def diff(self, other):
for t in Event.terms:
-if not self.has_key(t) or not other.has_key(t):
+if t not in self or t not in other:
continue
if not data_equal(self[t], other[t]):
log.warning("expected %s=%s, got %s" % (t, self[t], other[t]))
@@ -133,7 +139,7 @@ class Event(dict):
# - expected values assignments
class Test(object):
def __init__(self, path, options):
-parser = ConfigParser.SafeConfigParser()
+parser = configparser.SafeConfigParser()
parser.read(path)
log.warning("running '%s'" % path)
@@ -192,7 +198,7 @@ class Test(object):
return True
def load_events(self, path, events):
-parser_event = ConfigParser.SafeConfigParser()
+parser_event = configparser.SafeConfigParser()
parser_event.read(path)
# The event record section header contains 'event' word,
@@ -206,7 +212,7 @@ class Test(object):
# Read parent event if there's any
if (':' in section):
base = section[section.index(':') + 1:]
-parser_base = ConfigParser.SafeConfigParser()
+parser_base = configparser.SafeConfigParser()
parser_base.read(self.test_dir + '/' + base)
base_items = parser_base.items('event')
@@ -321,9 +327,9 @@ def run_tests(options):
for f in glob.glob(options.test_dir + '/' + options.test):
try:
Test(f, options).run()
-except Unsup, obj:
+except Unsup as obj:
log.warning("unsupp %s" % obj.getMsg())
-except Notest, obj:
+except Notest as obj:
log.warning("skipped %s" % obj.getMsg())
def setup_log(verbose):
@@ -362,7 +368,7 @@ def main():
parser.add_option("-p", "--perf",
action="store", type="string", dest="perf")
parser.add_option("-v", "--verbose",
- action="count", dest="verbose")
+ default=0, action="count", dest="verbose")
options, args = parser.parse_args()
if args:
@@ -372,7 +378,7 @@ def main():
setup_log(options.verbose)
if not options.test_dir:
-print 'FAILED no -d option specified'
+print('FAILED no -d option specified')
sys.exit(-1)
if not options.test:
@@ -381,8 +387,8 @@ def main():
try:
run_tests(options)
-except Fail, obj:
-print "FAILED %s" % obj.getMsg();
+except Fail as obj:
+print("FAILED %s" % obj.getMsg())
sys.exit(-1)
sys.exit(0)
--
2.20.1
[PATCH 3/6] perf script python: remove explicit shebang from setup.py
Makefile.perf invokes setup.py via an explicit invocation of python (PYTHON_WORD) so there is therefore no need for an explicit shebang. Also most distros follow pep-0394 which recommends that /usr/bin/python refer only to v2 and so may not exist on the system (if PYTHON=python3). Signed-off-By: Tony Jones Cc: Arnaldo Carvalho de Melo --- tools/perf/util/setup.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/tools/perf/util/setup.py b/tools/perf/util/setup.py index 64d1f36dee99..d3ffc18424b5 100644 --- a/tools/perf/util/setup.py +++ b/tools/perf/util/setup.py @@ -1,5 +1,3 @@ -#!/usr/bin/python - from os import getenv from subprocess import Popen, PIPE from re import sub -- 2.20.1
[PATCH 5/6] perf script python: remove explicit shebang from Python scripts
The scripts in scripts/python are intended to be run from 'perf script' and
the Python version used is dictated by how perf was built (PYTHON=).
Also most distros follow pep-0394 which recommends that /usr/bin/python
refer to Python2 and so may not exist on the system (if PYTHON=python3).
- Remove the explicit shebang
- Install the scripts as mode 644
Signed-off-by: Tony Jones
Cc: Frederic Weisbecker
Cc: Adrian Hunter
Cc: Jiri Olsa
---
tools/perf/Makefile.perf | 4 ++--
tools/perf/scripts/python/exported-sql-viewer.py | 1 -
tools/perf/scripts/python/sched-migration.py | 2 --
tools/perf/scripts/python/stat-cpi.py| 1 -
4 files changed, 2 insertions(+), 6 deletions(-)
diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index 0ee6795d82cc..09df1c8a4ec9 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -863,8 +863,8 @@ ifndef NO_LIBPYTHON
$(call QUIET_INSTALL, python-scripts) \
$(INSTALL) -d -m 755
'$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python/Perf-Trace-Util/lib/Perf/Trace';
\
$(INSTALL) -d -m 755
'$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python/bin'; \
- $(INSTALL) scripts/python/Perf-Trace-Util/lib/Perf/Trace/* -t
'$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python/Perf-Trace-Util/lib/Perf/Trace';
\
- $(INSTALL) scripts/python/*.py -t
'$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python'; \
+ $(INSTALL) scripts/python/Perf-Trace-Util/lib/Perf/Trace/* -m
644 -t
'$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python/Perf-Trace-Util/lib/Perf/Trace';
\
+ $(INSTALL) scripts/python/*.py -m 644 -t
'$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python'; \
$(INSTALL) scripts/python/bin/* -t
'$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python/bin'
endif
$(call QUIET_INSTALL, perf_completion-script) \
diff --git a/tools/perf/scripts/python/exported-sql-viewer.py
b/tools/perf/scripts/python/exported-sql-viewer.py
index f278ce5ebab7..c3091401df91 100755
--- a/tools/perf/scripts/python/exported-sql-viewer.py
+++ b/tools/perf/scripts/python/exported-sql-viewer.py
@@ -1,4 +1,3 @@
-#!/usr/bin/python2
# SPDX-License-Identifier: GPL-2.0
# exported-sql-viewer.py: view data from sql database
# Copyright (c) 2014-2018, Intel Corporation.
diff --git a/tools/perf/scripts/python/sched-migration.py
b/tools/perf/scripts/python/sched-migration.py
index 3473e7f66081..3984bf51f3c5 100644
--- a/tools/perf/scripts/python/sched-migration.py
+++ b/tools/perf/scripts/python/sched-migration.py
@@ -1,5 +1,3 @@
-#!/usr/bin/python
-#
# Cpu task migration overview toy
#
# Copyright (C) 2010 Frederic Weisbecker
diff --git a/tools/perf/scripts/python/stat-cpi.py
b/tools/perf/scripts/python/stat-cpi.py
index 8410672efb8b..a81ad8835a74 100644
--- a/tools/perf/scripts/python/stat-cpi.py
+++ b/tools/perf/scripts/python/stat-cpi.py
@@ -1,4 +1,3 @@
-#!/usr/bin/env python
# SPDX-License-Identifier: GPL-2.0
data= {}
--
2.20.1
[PATCH 1/6] perf script python: Add trace_context extension module to sys,modules
In Python3, the result of PyModule_Create (called from
scripts/python/Perf-Trace-Util/Context.c) is not automatically added to
sys.modules. See: https://bugs.python.org/issue4592
Below is the observed behavior without the fix:
# ldd /usr/bin/perf | grep -i python
libpython3.6m.so.1.0 => /usr/lib64/libpython3.6m.so.1.0
(0x7f8e1dfb2000)
# perf record /bin/false
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.015 MB perf.data (17 samples) ]
# perf script -g python | cat
generated Python script: perf-script.py
# perf script -s ./perf-script.py
Traceback (most recent call last):
File "./perf-script.py", line 18, in
from perf_trace_context import *
ModuleNotFoundError: No module named 'perf_trace_context'
Error running python script ./perf-script.py
Signed-off-By: Tony Jones
Cc: Jaroslav Škarvada
Fixes: 66dfdff03d196e51322c6a85c0d8db8bb2bdd655
---
tools/perf/util/scripting-engines/trace-event-python.c | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c
b/tools/perf/util/scripting-engines/trace-event-python.c
index 87ef16a1b17e..40300d8e80a7 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -1494,6 +1494,7 @@ static void _free_command_line(wchar_t **command_line,
int num)
static int python_start_script(const char *script, int argc, const char **argv)
{
struct tables *tables = _global;
+ PyMODINIT_FUNC (*initfunc)(void);
#if PY_MAJOR_VERSION < 3
const char **command_line;
#else
@@ -1504,24 +1505,25 @@ static int python_start_script(const char *script, int
argc, const char **argv)
FILE *fp;
#if PY_MAJOR_VERSION < 3
+ initfunc = initperf_trace_context;
command_line = malloc((argc + 1) * sizeof(const char *));
command_line[0] = script;
for (i = 1; i < argc + 1; i++)
command_line[i] = argv[i - 1];
#else
+ initfunc = PyInit_perf_trace_context;
command_line = malloc((argc + 1) * sizeof(wchar_t *));
command_line[0] = Py_DecodeLocale(script, NULL);
for (i = 1; i < argc + 1; i++)
command_line[i] = Py_DecodeLocale(argv[i - 1], NULL);
#endif
+ PyImport_AppendInittab("perf_trace_context", initfunc);
Py_Initialize();
#if PY_MAJOR_VERSION < 3
- initperf_trace_context();
PySys_SetArgv(argc + 1, (char **)command_line);
#else
- PyInit_perf_trace_context();
PySys_SetArgv(argc + 1, command_line);
#endif
--
2.20.1
[PATCH 0/6] Fix issues with Python3 scripting
Seeteena posted, earlier this week, some patches to add Python3 support to scripts/python/*.py. Unfortunately there were some issues with these patches (such as: https://lkml.org/lkml/2019/1/17/351) Since I already had a tested set of patches in openSUSE:Factory and SLE15-SP1 and was about to submit them, Seeteena and I that agreed I should post my patches not involving scripts/python/*.py and Seeteena will later resubmit the patches for scripts/python/*.py incorporating my review feedback under a joint signed-off-by. It should be noted that the use of "from __future__ import print_function" (see: https://lkml.org/lkml/2019/1/16/641) and "except as" (see change to: tests/attr.py) implies Python2 >= 2.6 as the necessary support has not been backported to prior versions. I am not sure if it's worth detecting <2.6 at build time or whether it's sufficiently old as to be a non-issue? The shebang changes were driven mostly by our build process as it scans all files within an rpm and the shebangs would result in a rpm requires on the python2 binary when BuildRequires was python3-devel. I think they make sense to apply upstream but understand totally if it's prefered we keep them local. These changes have been tested with PYTHON=python2 (v2.7) and PYTHON=python3 (v3.6) on latest openSUSE Tumbleweed. I did notice that test #18 "'import perf' in python" is failing on my system without these changes. I'll look at it further but didn't want to hold up Seeteena's resubmit. Tony Jones (6): perf script python: Add trace_context extension module to sys,modules perf script python: Use PyBytes for attr in trace-event-python perf script python: remove explicit shebang from setup.py perf script python: remove explicit shebang from tests/attr.c perf script python: remove explicit shebang from Python scripts perf script python: add Python3 support to tests/attr.py tools/perf/Makefile.perf | 4 +-- .../scripts/python/exported-sql-viewer.py | 1 - tools/perf/scripts/python/sched-migration.py | 2 -- tools/perf/scripts/python/stat-cpi.py | 1 - tools/perf/tests/attr.py | 33 +++ .../scripting-engines/trace-event-python.c| 9 ++--- tools/perf/util/setup.py | 2 -- 7 files changed, 26 insertions(+), 26 deletions(-) -- 2.20.1
[PATCH 2/6] perf script python: Use PyBytes for attr in trace-event-python
With Python3. PyUnicode_FromStringAndSize is unsafe to call on attr and will
return NULL. Use _PyBytes_FromStringAndSize (as with raw_buf).
Below is the observed behavior without the fix. Note it is first necessary
to apply the prior fix (Add trace_context extension module to sys,modules):
# ldd /usr/bin/perf | grep -i python
libpython3.6m.so.1.0 => /usr/lib64/libpython3.6m.so.1.0
(0x7f8e1dfb2000)
# perf record -e raw_syscalls:sys_enter /bin/false
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.018 MB perf.data (21 samples) ]
# perf script -g python | cat
generated Python script: perf-script.py
# perf script -s ./perf-script.py
in trace_begin
Segmentation fault (core dumped)
Signed-off-By: Tony Jones
Cc: Jaroslav Škarvada
Fixes: 66dfdff03d196e51322c6a85c0d8db8bb2bdd655
---
tools/perf/util/scripting-engines/trace-event-python.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c
b/tools/perf/util/scripting-engines/trace-event-python.c
index 40300d8e80a7..777c97b13905 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -733,8 +733,7 @@ static PyObject *get_perf_sample_dict(struct perf_sample
*sample,
Py_FatalError("couldn't create Python dictionary");
pydict_set_item_string_decref(dict, "ev_name",
_PyUnicode_FromString(perf_evsel__name(evsel)));
- pydict_set_item_string_decref(dict, "attr",
_PyUnicode_FromStringAndSize(
- (const char *)>attr, sizeof(evsel->attr)));
+ pydict_set_item_string_decref(dict, "attr",
_PyBytes_FromStringAndSize((const char *)>attr, sizeof(evsel->attr)));
pydict_set_item_string_decref(dict_sample, "pid",
_PyLong_FromLong(sample->pid));
--
2.20.1
[tip:perf/urgent] perf script: Fix crash when processing recorded stat data
Commit-ID: 8bf8c6da53c2265aea365a1de6038f118f522113
Gitweb: https://git.kernel.org/tip/8bf8c6da53c2265aea365a1de6038f118f522113
Author: Tony Jones
AuthorDate: Sun, 20 Jan 2019 11:14:14 -0800
Committer: Arnaldo Carvalho de Melo
CommitDate: Mon, 21 Jan 2019 11:29:07 -0300
perf script: Fix crash when processing recorded stat data
While updating perf to work with Python3 and Python2 I noticed that the
stat-cpi script was dumping core.
$ perf stat -e cycles,instructions record -o /tmp/perf.data /bin/false
Performance counter stats for '/bin/false':
802,148 cycles
604,622 instructions
802,148 cycles
604,622 instructions
0.001445842 seconds time elapsed
$ perf script -i /tmp/perf.data -s scripts/python/stat-cpi.py
Segmentation fault (core dumped)
...
...
rblist=rblist@entry=0xb2a200 ,
new_entry=new_entry@entry=0x7ffcb755c310) at util/rblist.c:33
ctx=, type=, create=,
cpu=, evsel=) at util/stat-shadow.c:118
ctx=, type=, st=)
at util/stat-shadow.c:196
count=count@entry=727442, cpu=cpu@entry=0, st=0xb2a200 )
at util/stat-shadow.c:239
config=config@entry=0xafeb40 ,
counter=counter@entry=0x133c6e0) at util/stat.c:372
...
...
The issue is that since 1fcd03946b52 perf_stat__update_shadow_stats now calls
update_runtime_stat passing rt_stat rather than calling update_stats but
perf_stat__init_shadow_stats has never been called to initialize rt_stat in
the script path processing recorded stat data.
Since I can't see any reason why perf_stat__init_shadow_stats() is presently
initialized like it is in builtin-script.c::perf_sample__fprint_metric()
[4bd1bef8bba2f] I'm proposing it instead be initialized once in __cmd_script
Committer testing:
After applying the patch:
# perf script -i /tmp/perf.data -s tools/perf/scripts/python/stat-cpi.py
0.001970: cpu -1, thread -1 -> cpi 1.709079 (1075684/629394)
#
No segfault.
Signed-off-by: Tony Jones
Reviewed-by: Jiri Olsa
Tested-by: Arnaldo Carvalho de Melo
Tested-by: Ravi Bangoria
Cc: Andi Kleen
Cc: Jin Yao
Fixes: 1fcd03946b52 ("perf stat: Update per-thread shadow stats")
Link: http://lkml.kernel.org/r/20190120191414.12925-1-to...@suse.de
Signed-off-by: Arnaldo Carvalho de Melo
---
tools/perf/builtin-script.c | 7 ++-
1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 357906ed1898..ac221f137ed2 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -1681,13 +1681,8 @@ static void perf_sample__fprint_metric(struct
perf_script *script,
.force_header = false,
};
struct perf_evsel *ev2;
- static bool init;
u64 val;
- if (!init) {
- perf_stat__init_shadow_stats();
- init = true;
- }
if (!evsel->stats)
perf_evlist__alloc_stats(script->session->evlist, false);
if (evsel_script(evsel->leader)->gnum++ == 0)
@@ -2359,6 +2354,8 @@ static int __cmd_script(struct perf_script *script)
signal(SIGINT, sig_handler);
+ perf_stat__init_shadow_stats();
+
/* override event processing functions */
if (script->show_task_events) {
script->tool.comm = process_comm_event;
Re: [PATCH] perf scripts python: Add Python 3 support to mem-phys-addr.py
On 1/17/19 1:48 AM, seeteena wrote: > I have added >> You have not added "from __future__ import print_function", so you're >> relying on a Python 2 parsing oddity to make this work. I didn't see Jon's comment earlier. Sorry to have given conflicting advice. I'd not considered it a parsing oddity but I think he's correct. There is a flip side however and that is that this oddity probably works in versions older than v2.6 which is as far back as future support exists. I can't imagine anyone is using < 2.6 but I have no idea what PYTHON=python2 actually implies. The other option is to switch to sys.write.stdout which has always had more consistent semantics. Tony
Re: [PATCH v3] perf scripts python: Add Python 3 support to export-to-sqlite.py
On 1/21/19 2:39 AM, Seeteena Thoufeek wrote:
> Support both Python 2 and Python 3 in export-to-sqlite.py. ``print`` is
> now a function rather than a statement. This should have no functional
> change.
>
> Also, handles the conversion of "print >> sys.stderr".
>
> Signed-off-by: Seeteena Thoufeek
> Reviewed-by: Ravi Bangoria
> ---
> tools/perf/scripts/python/export-to-sqlite.py | 19 ++-
> 1 file changed, 10 insertions(+), 9 deletions(-)
>
> diff --git a/tools/perf/scripts/python/export-to-sqlite.py
> b/tools/perf/scripts/python/export-to-sqlite.py
> index 245caf2..95bfcb2 100644
> --- a/tools/perf/scripts/python/export-to-sqlite.py
> +++ b/tools/perf/scripts/python/export-to-sqlite.py
> @@ -9,6 +9,7 @@
> # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
> # more details.
> +from __future__ import print_function
So is the script failing for you without the above
As I said in previous reviews you should not need this unless you are using
features of the latest print function (end, sep etc). You will need this import
in the other scripts (once you fix the numerous errors regarding the trailing
comma usage) as some need to concatenate output onto one line but I don't think
you need it here.
tony
$ python2 --version
Python 2.7.15
$ python3 --version
Python 3.6.5
$ a='print("abcd") ; print("defg")'
$ echo $a | python2
abcd
defg
$ echo $a | python3
abcd
defg
$ b='print("abcd", end="") ; print("defg")'
$ echo $b | python2
File "", line 1
print("abcd", end="") ; print("defg")
^
SyntaxError: invalid syntax
$ echo $b | python3
abcddefg
$ c='from __future__ import print_function ; print("abcd", end="") ;
print("defg")'
$ echo $c | python2
abcddefg
$ echo $c | python3
abcddefg
Re: [PATCH v2] perf scripts python: Add Python 3 support to exported-sql-viewer.py
On 1/20/19 11:27 AM, Jonathan Corbet wrote: > On Fri, 18 Jan 2019 16:45:04 -0800 > Tony Jones wrote: > >> On 1/17/19 1:45 AM, Seeteena Thoufeek wrote: >> >>> +if sys.version_info[0] < 3: >>> +import cPickle >>> +else: >>> +import _pickle as cPickle >> >> Do you really need this? >> >> pickle is already in Python2. > > Did you mean in Python3? I would agree that using it is better than > importing the semi-hidden _pickle module. No. I meant Python2 :) pickle in Python2 is the python implementation cPickle in Python2 is the C implementation. Read: https://docs.python.org/3.1/whatsnew/3.0.html#library-changes A common pattern in Python 2.x is to have one version of a module implemented in pure Python, with an optional accelerated version implemented as a C extension; for example, pickle and cPickle. This places the burden of importing the accelerated version and falling back on the pure Python version on each user of these modules. In Python 3.0, the accelerated versions are considered implementation details of the pure Python versions. Users should always import the standard version, which attempts to import the accelerated version and falls back to the pure Python version. I my patchset "import pickle" was sufficient for Python2 and Python3 The question I suppose is whether this script, for Python2, needs the accelerated C implementation. I decided it didn't. Tony
[PATCH] perf script: fix crash when processing recorded stat data
While updating Perf to work with Python3 and Python2 I noticed that the
stat-cpi script was dumping core.
$ perf stat -e cycles,instructions record -o /tmp/perf.data /bin/false
Performance counter stats for '/bin/false':
802,148 cycles
604,622 instructions
802,148 cycles
604,622 instructions
0.001445842 seconds time elapsed
$ perf script -i /tmp/perf.data -s scripts/python/stat-cpi.py
Segmentation fault (core dumped)
...
...
rblist=rblist@entry=0xb2a200 ,
new_entry=new_entry@entry=0x7ffcb755c310) at util/rblist.c:33
ctx=, type=, create=,
cpu=, evsel=) at util/stat-shadow.c:118
ctx=, type=, st=)
at util/stat-shadow.c:196
count=count@entry=727442, cpu=cpu@entry=0, st=0xb2a200 )
at util/stat-shadow.c:239
config=config@entry=0xafeb40 ,
counter=counter@entry=0x133c6e0) at util/stat.c:372
...
...
The issue is that since 1fcd03946b52 perf_stat__update_shadow_stats now calls
update_runtime_stat passing rt_stat rather than calling update_stats but
perf_stat__init_shadow_stats has never been called to initialize rt_stat in
the script path processing recorded stat data.
Since I can't see any reason why perf_stat__init_shadow_stats() is presently
initialized like it is in builtin-script.c::perf_sample__fprint_metric()
[4bd1bef8bba2f] I'm proposing it instead be initialized once in __cmd_script
Fixes: 1fcd03946b52 ("perf stat: Update per-thread shadow stats")
Signed-off-by: Tony Jones
---
tools/perf/builtin-script.c | 7 ++-
1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index d079f36d342d..9a6dd86e606f 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -1681,13 +1681,8 @@ static void perf_sample__fprint_metric(struct
perf_script *script,
.force_header = false,
};
struct perf_evsel *ev2;
- static bool init;
u64 val;
- if (!init) {
- perf_stat__init_shadow_stats();
- init = true;
- }
if (!evsel->stats)
perf_evlist__alloc_stats(script->session->evlist, false);
if (evsel_script(evsel->leader)->gnum++ == 0)
@@ -2359,6 +2354,8 @@ static int __cmd_script(struct perf_script *script)
signal(SIGINT, sig_handler);
+ perf_stat__init_shadow_stats();
+
/* override event processing functions */
if (script->show_task_events) {
script->tool.comm = process_comm_event;
--
2.18.0
Re: [PATCH v2] perf scripts python: Add Python 3 support to failed-syscalls-by-pid.py
On 1/18/19 5:12 PM, Tony Jones wrote: > On 1/17/19 1:45 AM, Seeteena Thoufeek wrote: > >> +from __future__ import print_function > > You don't need this unless you're actually requiring functionality that only > exists in v3. > For example, you need it to handle the suppress newline functionality such as > "end=". Also, it brings up a question I had. What python versions are expected to be supported with PYTHON=python2? https://python-future.org/imports.html So importing from future (for end='' support) implies >= 2.6. The base release of 2.6 was October 2008 so maybe this is fine. Otherwise it may be preferable to use sys.stdout.write which has more consistent semantics (see tools/perf/scripts/python/compaction-times.py) Tony
Re: [PATCH v2] perf scripts python: Add Python 3 support to check-perf-trace.py
On 1/18/19 4:29 PM, Tony Jones wrote:
> I'd been simultaneously working on a patch set to fix up Python3.
>
> It's actually already in our Factory and SLE15-SP1 releases as we had a
> deadline to kill Python2 usage for internal rpms.
>
> I was going to post once I'd fixed the last remaining issue ('import perf' is
> still failing [test #18]).
>
> I guess "you snooze you lose" :-)
>
> https://build.opensuse.org/package/view_file/devel:tools/perf/perf.changes?expand=1
Seeteena, I'm than happy to forward my patches via email. Alternatively, as I
said in another post, the full series is in SLE15-SP1/beta2 (should apply to
tip) which IBM has access to and they've been backported to v4.19 (Factory) at
https://build.opensuse.org/package/show/openSUSE:Factory/perf. Everything has
been tested against python2 and python3.
Also, in this series there are patches to:
- port tests/attr.py to Python3
- remove shebangs from the .py scripts and change to mode 644 as IMO it makes
no sense to explicitly have #!/usr/bin/python since per pep-0394 this refers to
version2 and the system may only have python3 installed
- remove the shebang from setup.py since it's explicitly invoked via call to
${PYTHON_WORD}
Also I found in testing that the following fix is also needed. I'm not 100%
sure on it and was going to revisit before posting but
_PyUnicode_FromStringAndSize() is definitely unsafe to use on attr.
Fixes: 66dfdff03d196e51322c6a85c0d8db8bb2bdd655
With Python3. PyUnicode_FromStringAndSize is unsafe to call on attr and will
return NULL. Use _PyBytes_FromStringAndSize (as with raw_buf).
$ perf script -s perf-script.py -i perf.data
in trace_begin
Segmentation fault (core dumped)
---
tools/perf/util/scripting-engines/trace-event-python.c |3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -733,8 +733,7 @@ static PyObject *get_perf_sample_dict(st
Py_FatalError("couldn't create Python dictionary");
pydict_set_item_string_decref(dict, "ev_name",
_PyUnicode_FromString(perf_evsel__name(evsel)));
- pydict_set_item_string_decref(dict, "attr",
_PyUnicode_FromStringAndSize(
- (const char *)>attr, sizeof(evsel->attr)));
+ pydict_set_item_string_decref(dict, "attr",
_PyBytes_FromStringAndSize((const char *)>attr, sizeof(evsel->attr)));
pydict_set_item_string_decref(dict_sample, "pid",
_PyLong_FromLong(sample->pid));
Re: [PATCH v2] perf scripts python: Add Python 3 support to stackcollapse.py
On 1/17/19 1:45 AM, Seeteena Thoufeek wrote:
> Support both Python 2 and Python 3 in stackcollapse.py. ``print`` is now a
> function rather than a statement. This should have no functional change.
>
> Signed-off-by: Seeteena Thoufeek
> Reviewed-by: Ravi Bangoria
> ---
> tools/perf/scripts/python/stackcollapse.py | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/tools/perf/scripts/python/stackcollapse.py
> b/tools/perf/scripts/python/stackcollapse.py
> index 1697b5e..f77bc0d 100755
> --- a/tools/perf/scripts/python/stackcollapse.py
> +++ b/tools/perf/scripts/python/stackcollapse.py
> @@ -18,6 +18,7 @@
> #
> # Written by Paolo Bonzini
> # Based on Brendan Gregg's stackcollapse-perf.pl script.
> +from __future__ import print_function
Again, not necessary.
> import os
> import sys
> @@ -123,4 +124,4 @@ def trace_end():
> list = lines.keys()
> list.sort()
> for stack in list:
> -print "%s %d" % (stack, lines[stack])
> +print("%s %d" % (stack, lines[stack]))
>
Did you test any of these changes with Python3?
If you run 'ldd /usr/bin/perf | grep python' when you've built with
PYTHON=python3, what do you see?
$ ldd /usr/bin/perf | grep python
libpython3.6m.so.1.0 => /usr/lib64/libpython3.6m.so.1.0
(0x7fea66701000)
If you do, you'll run into this error:
$ /usr/bin/perf script -s scripts/python/stackcollapse.py -i /tmp/perf.data
Traceback (most recent call last):
File "scripts/python/stackcollapse.py", line 124, in trace_end
list.sort()
AttributeError: 'dict_keys' object has no attribute 'sort'
Fatal Python error: problem in Python trace event handler
You need the following change in addition:
--- a/tools/perf/scripts/python/stackcollapse.py
+++ b/tools/perf/scripts/python/stackcollapse.py
@@ -120,7 +120,6 @@ def process_event(param_dict):
lines[stack_string] = lines[stack_string] + 1
def trace_end():
-list = lines.keys()
-list.sort()
+list = sorted(lines)
for stack in list:
-print "%s %d" % (stack, lines[stack])
+print ("%s %d" % (stack, lines[stack]))
As I said in a different post, I'd not yet posted my series as I was still
trying to fix the failure of the "import perf" testcase (aka tools/perf/python)
and (apologies) I'd missed your V1 but maybe I should post them as they've been
thoroughly integrated into perf and tested with python2 and python3.
The full patchset is in SLE15-SP1 beta2 which I know IBM has access to. A
backport is also in Factory.
Tony
Re: [PATCH v2] perf scripts python: Add Python 3 support to sctop.py
On 1/17/19 1:45 AM, Seeteena Thoufeek wrote:
> +from __future__ import print_function
Again, you don't need this.
> - print "\nsyscall events for %s:\n\n" % (for_comm),
> + print("\nsyscall events for %s:\n\n" % (for_comm)),
same comments regarding trailing comma usage
$ git annotate tools/perf/scripts/python/sctop.py | grep thread
2e7d1e3fb8043 (Arnaldo Carvalho de Melo 2010-10-25 18:39:20 -0200
11)import os, sys, thread, time
47902f3611b39 (Tom Zanussi2010-04-01 23:59:23 -0500 42)
thread.start_new_thread(print_syscall_totals, (interval,))
The low level threading api has changed in the most recent Python versions. I
think you'll find you also need:
@@ -8,7 +8,12 @@
# will be refreshed every [interval] seconds. The default interval is
# 3 seconds.
-import os, sys, thread, time
+import os, sys, time
+
+try:
+import thread
+except ImportError:
+import _thread as thread
Tony
Re: [PATCH v2] perf scripts python: Add Python 3 support to netdev-times.py
On 1/17/19 1:45 AM, Seeteena Thoufeek wrote:> Support both Python 2 and Python 3 in netdev-times.py. ``print`` > is now a function rather than a statement. This should have no > functional change. Same feedback as all the other patches applies. In addition: $ git annotate tools/perf/scripts/python/netdev-times.py | grep "all_event_list.sort" 359d5106a2ff4 (Koki Sanagi2010-08-23 18:47:09 +0900 175) all_event_list.sort(lambda a,b :cmp(a[EINFO_IDX_TIME], I didn't think the above was valid with Python3. In my version (https://build.opensuse.org/package/view_file/devel:tools/perf/port-failed-syscalls-by-pid-script-to-python3.patch?expand=1) I reworked it as: from functools import cmp_to_key all_event_list.sort(key=cmp_to_key(lambda a,b :a[EINFO_IDX_TIME] < b[EINFO_IDX_TIME])) Tony
Re: [PATCH v2] perf scripts python: Add Python 3 support to intel-pt-events.py
On 1/17/19 1:45 AM, Seeteena Thoufeek wrote:
> Support both Python 2 and Python 3 in intel-pt-events.py.``print``
> is now a function rather than a statement. This should have no
> functional change.
>
> Fixes indentation issue, replace spaces with tab.
Again, trailing comma use is incorrect.
> - print "%16s %5u/%-5u [%03u] %9u.%09u %7s:" % (comm, pid, tid, cpu, ts /
> 10, ts %10, name),
> + print("%16s %5u/%-5u [%03u] %9u.%09u %7s:" % (comm, pid, tid, cpu, ts /
> 10, ts %10, name)),
print ("%16s %5u/%-5u [%03u] %9u.%09u %7s:" % (comm, pid, tid, cpu, ts /
10, ts %10, name), end='')
and so on for rest of changes.
Tony
Re: [PATCH v2] perf scripts python: Add Python 3 support to failed-syscalls-by-pid.py
On 1/17/19 1:45 AM, Seeteena Thoufeek wrote:
> +from __future__ import print_function
You don't need this unless you're actually requiring functionality that only
exists in v3.
For example, you need it to handle the suppress newline functionality such as
"end=".
> def print_error_totals():
> if for_comm is not None:
> - print "\nsyscall errors for %s:\n\n" % (for_comm),
> + print("\nsyscall errors for %s:\n\n" % (for_comm)),
> else:
> - print "\nsyscall errors:\n\n",
> + print("\nsyscall errors:\n\n"),
>
> -print "%-30s %10s\n" % ("comm [pid]", "count"),
> -print "%-30s %10s\n" % ("--", \
> - "--"),
> +print("%-30s %10s\n" % ("comm [pid]", "count")),
> +print("%-30s %10s\n" % ("--", \
> + "--")),
Same comments as before regarding trailing comma for function.
See:
https://build.opensuse.org/package/view_file/devel:tools/perf/port-failed-syscalls-by-pid-script-to-python3.patch?expand=1
Re: [PATCH v2] perf scripts python: Add Python 3 support to export-to-sqlite.py
On 1/17/19 1:45 AM, Seeteena Thoufeek wrote:
> Support both Python 2 and Python 3 in export-to-sqlite.py. ``print`` is
> now a function rather than a statement. This should have no functional
> change.
I don't see any changes handling the following:
$ git annotate tools/perf/scripts/python/export-to-sqlite.py | grep ">> sys"
564b9527d1ccf (Adrian Hunter 2017-08-03 11:31:28 +0300 64) print
>> sys.stderr, "Usage is: export-to-sqlite.py []
[] []"
564b9527d1ccf (Adrian Hunter 2017-08-03 11:31:28 +0300 65) print
>> sys.stderr, "where:columns 'all' or 'branches'"
564b9527d1ccf (Adrian Hunter 2017-08-03 11:31:28 +0300 66) print
>> sys.stderr, " calls 'calls' => create calls and
call_paths table"
564b9527d1ccf (Adrian Hunter 2017-08-03 11:31:28 +0300 67) print
>> sys.stderr, " callchains 'callchains' => create call_paths
table"
$ echo 'import sys ; print >> sys.stderr, "Usage is: exported-sql-viewer.py
{ | --help-only}"' | python2
Usage is: exported-sql-viewer.py { | --help-only}
$ echo 'import sys ; print >> sys.stderr, "Usage is: exported-sql-viewer.py
{ | --help-only}"' | python3
Traceback (most recent call last):
File "", line 1, in
TypeError: unsupported operand type(s) for >>: 'builtin_function_or_method' and
'_io.TextIOWrapper'. Did you mean "print(, file=)"?
They are best handled via conversion to sys.stderr.write() since sys is already
imported.
Tony
Re: [PATCH v2] perf scripts python: Add Python 3 support to check-perf-trace.py
On 1/17/19 1:45 AM, Seeteena Thoufeek wrote:
> - print "vec=%s\n" % \
> - (symbol_str("irq__softirq_entry", "vec", vec)),
> + print("vec=%s\n" % \
> + (symbol_str("irq__softirq_entry", "vec", vec))),
Again, check the trailing comma usage:
$ echo 'print "abc", ; print "def"' | python2
abc def
$ echo 'print ("abc"), ; print ("def")' | python2
abc def
$ echo 'print ("abc"), ; print ("def")' | python3
abc
def
Re: [PATCH v2] perf scripts python: Add Python 3 support to syscall-counts-by-pid.py
On 1/17/19 1:45 AM, Seeteena Thoufeek wrote:
> - print "\nsyscall events for %s:\n\n" % (for_comm),
> + print("\nsyscall events for %s:\n\n" % (for_comm)),
> else:
> - print "\nsyscall events by comm/pid:\n\n",
> + print("\nsyscall events by comm/pid:\n\n"),
>
> -print "%-40s %10s\n" % ("comm [pid]/syscalls", "count"),
> -print "%-40s %10s\n" % ("", \
> - "--"),
> +print("%-40s %10s\n" % ("comm [pid]/syscalls", "count")),
> +print("%-40s %10s\n" % ("", \
> + "--")),
Is the 'print (x),' [trailing comma] syntax valid for function syntax?
Print "x", in Py2 means suppress the trailing newline.
You need to actually run the scripts (old, new PYTHON=python2, new
PYTHON=python3) and compare the output.
This:
print "%-40s %10s\n" % ("comm [pid]/syscalls", "count"),
can be reworked as:
print ("%-40s %10s" % ("comm [pid]/syscalls", "count"))
See:
https://build.opensuse.org/package/view_file/devel:tools/perf/port-failed-syscalls-by-pid-script-to-python3.patch?expand=1
Re: [PATCH v2] perf scripts python: Add Python 3 support to exported-sql-viewer.py
On 1/17/19 1:45 AM, Seeteena Thoufeek wrote:
> +if sys.version_info[0] < 3:
> +import cPickle
> +else:
> +import _pickle as cPickle
Do you really need this?
pickle is already in Python2.
Also, did you test these changes on Python3?
I think you'll find you also need the following hunk otherwise you're running
into Unicode diffs on Py3.
@@ -2590,8 +2590,8 @@ def Main():
is_sqlite3 = False
try:
- f = open(dbname)
- if f.read(15) == "SQLite format 3":
+ f = open(dbname, "rb")
+ if f.read(15) == b'SQLite format 3':
is_sqlite3 = True
f.close()
except:
Plus you need to handle the conversion of "print >> sys.stderr"
Attached is my version (against tip):
--- a/tools/perf/scripts/python/exported-sql-viewer.py
+++ b/tools/perf/scripts/python/exported-sql-viewer.py
@@ -91,7 +91,7 @@ import sys
import weakref
import threading
import string
-import cPickle
+import pickle
import re
import os
from PySide.QtCore import *
@@ -1559,7 +1559,7 @@ class SQLTableDialogDataItem():
return str(lower_id)
def ConvertRelativeTime(self, val):
- print "val ", val
+ print("val ", val)
mult = 1
suffix = val[-2:]
if suffix == "ms":
@@ -1581,29 +1581,29 @@ class SQLTableDialogDataItem():
return str(val)
def ConvertTimeRange(self, vrange):
- print "vrange ", vrange
+ print("vrange ", vrange)
if vrange[0] == "":
vrange[0] = str(self.first_time)
if vrange[1] == "":
vrange[1] = str(self.last_time)
vrange[0] = self.ConvertRelativeTime(vrange[0])
vrange[1] = self.ConvertRelativeTime(vrange[1])
- print "vrange2 ", vrange
+ print("vrange2 ", vrange)
if not self.IsNumber(vrange[0]) or not self.IsNumber(vrange[1]):
return False
- print "ok1"
+ print("ok1")
beg_range = max(int(vrange[0]), self.first_time)
end_range = min(int(vrange[1]), self.last_time)
if beg_range > self.last_time or end_range < self.first_time:
return False
- print "ok2"
+ print("ok2")
vrange[0] = self.BinarySearchTime(0, self.last_id, beg_range,
True)
vrange[1] = self.BinarySearchTime(1, self.last_id + 1,
end_range, False)
- print "vrange3 ", vrange
+ print("vrange3 ", vrange)
return True
def AddTimeRange(self, value, ranges):
- print "value ", value
+ print("value ", value)
n = value.count("-")
if n == 1:
pass
@@ -2577,7 +2577,7 @@ class DBRef():
def Main():
if (len(sys.argv) < 2):
- print >> sys.stderr, "Usage is: exported-sql-viewer.py
{ | --help-only}"
+ sys.stderr.write("Usage is: exported-sql-viewer.py { | --help-only}\n");
raise Exception("Too few arguments")
dbname = sys.argv[1]
@@ -2590,8 +2590,8 @@ def Main():
is_sqlite3 = False
try:
- f = open(dbname)
- if f.read(15) == "SQLite format 3":
+ f = open(dbname, "rb")
+ if f.read(15) == b'SQLite format 3':
is_sqlite3 = True
f.close()
except:
Re: [PATCH v2] perf scripts python: Add Python 3 support to check-perf-trace.py
On 1/17/19 4:32 AM, Jiri Olsa wrote:
> On Thu, Jan 17, 2019 at 03:15:28PM +0530, Seeteena Thoufeek wrote:
>> Support both Python 2 and Python 3 in check-perf-trace.py.
>> ``print`` is now a function rather than a statement. This should have
>> no functional change.
>>
>> Fix indentation issue, replace spaces with tab
>>
>> Signed-off-by: Seeteena Thoufeek
>> Reviewed-by: Ravi Bangoria
>
> hum, could you please add some info about testing those changes?
> (or even some global into 0/.. patch)
>
> this is working for me on python2:
>
> [root@krava perf]# perf script rec check-perf-trace
> ^C
> [root@krava perf]# perf script -s scripts/python/check-perf-trace.py
> trace_begin
>
> unhandled events:
>
>
> event count
>
> ---
>
> raw_syscalls__sys_enter 3509879
>
>
> but fails for python3:
>
> [root@ibm-x3650m4-01-vm-04 perf]# perf script rec check-perf-trace
> ^C[ perf record: Woken up 0 times to write data ]
> Warning:
> 1 out of order events recorded.
> [ perf record: Captured and wrote 43.132 MB perf.data (490171 samples) ]
>
> [root@ibm-x3650m4-01-vm-04 perf]# perf script -s
> scripts/python/check-perf-trace.py
> Traceback (most recent call last):
> File "scripts/python/check-perf-trace.py", line 18, in
> from perf_trace_context import *
> ModuleNotFoundError: No module named 'perf_trace_context'
> Error running python script scripts/python/check-perf-trace.py
>
> I did not test with rpm, just did 'make install' for perf
>
> thanks,
> jirka
>
I'd been simultaneously working on a patch set to fix up Python3.
It's actually already in our Factory and SLE15-SP1 releases as we had a
deadline to kill Python2 usage for internal rpms.
I was going to post once I'd fixed the last remaining issue ('import perf' is
still failing [test #18]).
I guess "you snooze you lose" :-)
https://build.opensuse.org/package/view_file/devel:tools/perf/perf.changes?expand=1
Anyhow, the fix for the above is:
'add-trace_context-extension-module-to-sys-modules.patch' from above.
Attached below. Verified with PYTHON=python2 and PYTHON=python3
Tony
-
In Python3, the result of PyModule_Create (called from
scripts/python/Perf-Trace-Util/Context.c) is not automatically added to
sys.modules. See: https://bugs.python.org/issue4592
Below is the observed behavior without the fix.
# ldd /usr/bin/perf | grep -i python
libpython3.6m.so.1.0 => /usr/lib64/libpython3.6m.so.1.0
(0x7f8e1dfb2000)
# perf record -a -e raw_syscalls:sys_enter sleep 5
[ perf record: Woken up 0 times to write data ]
[ perf record: Captured and wrote 187.177 MB perf.data (1581501 samples) ]
# perf script -g python | cat
generated Python script: perf-script.py
# perf script -s ./perf-script.py
Traceback (most recent call last):
File "./perf-script.py", line 18, in
from perf_trace_context import *
ModuleNotFoundError: No module named 'perf_trace_context'
Error running python script ./perf-script.py
Signed-off-by: Tony Jones
---
tools/perf/util/scripting-engines/trace-event-python.c |6 --
1 file changed, 4 insertions(+), 2 deletions(-)
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -1494,6 +1494,7 @@ static void _free_command_line(wchar_t *
static int python_start_script(const char *script, int argc, const char **argv)
{
struct tables *tables = _global;
+ PyMODINIT_FUNC (*initfunc)(void);
#if PY_MAJOR_VERSION < 3
const char **command_line;
#else
@@ -1504,24 +1505,25 @@ static int python_start_script(const cha
FILE *fp;
#if PY_MAJOR_VERSION < 3
+ initfunc = initperf_trace_context;
command_line = malloc((argc + 1) * sizeof(const char *));
command_line[0] = script;
for (i = 1; i < argc + 1; i++)
command_line[i] = argv[i - 1];
#else
+ initfunc = PyInit_perf_trace_context;
command_line = malloc((argc + 1) * sizeof(wchar_t *));
command_line[0] = Py_DecodeLocale(script, NULL);
for (i = 1; i < argc + 1; i++)
command_line[i] = Py_DecodeLocale(argv[i - 1], NULL);
#endif
+ PyImport_AppendInittab("perf_trace_context", initfunc);
Py_Initialize();
#if PY_MAJOR_VERSION < 3
- initperf_trace_context();
PySys_SetArgv(argc + 1, (char **)command_line);
#else
- PyInit_perf_trace_context();
PySys_SetArgv(argc + 1, command_line);
#endif
Re: [PATCH] apparmor: Fix network performance issue in aa_label_sk_perm
On 09/07/2018 09:37 AM, John Johansen wrote:
> hey Tony,
>
> thanks for the patch, I am curious did you're investigation look
> into what parts of DEFINE_AUDIT_SK are causing the issue?
Hi JJ.
Attached are the perf annotations for DEFINE_AUDIT_SK (percentages are relative
to the fn).
Our kernel performance testing is carried out with default installs which means
AppArmor
is enabled but the performance tests are unconfined. It was obvious that the
overhead of
DEFINE_AUDIT_SK was significant for smaller packet sizes (typical of synthetic
benchmarks)
and that it didn't need to execute for the unconfined case, hence the patch.
I didn't
spend any time looking at the performance of confined tasks. It may be worth
your time to
look at this.
Comparing my current tip (2601dd392dd1) to tip+patch I'm seeing an increase of
3-6% in netperf
throughput for packet sizes 64-1024.
HTH
Tony
Percent | Source code & Disassembly of vmlinux for cycles:ppp (117
samples)
-
:
:
:
: Disassembly of section .text:
:
: 813fbec0 :
: aa_label_sk_perm():
:
type));
: }
:
: static int aa_label_sk_perm(struct aa_label
*label, const char *op, u32 request,
: struct sock *sk)
: {
0.00 : 813fbec0: callq 81a017f0 <__fentry__>
2.56 : 813fbec5: push %r14
0.00 : 813fbec7: mov%rcx,%r14
: struct aa_profile *profile;
: DEFINE_AUDIT_SK(sa, op, sk);
0.00 : 813fbeca: mov$0x7,%ecx
: {
0.00 : 813fbecf: push %r13
3.42 : 813fbed1: mov%edx,%r13d
0.00 : 813fbed4: push %r12
0.00 : 813fbed6: push %rbp
0.00 : 813fbed7: mov%rdi,%rbp
5.13 : 813fbeda: push %rbx
0.00 : 813fbedb: sub$0xb8,%rsp
: DEFINE_AUDIT_SK(sa, op, sk);
0.00 : 813fbee2: movzwl 0x10(%r14),%r9d
: {
1.71 : 813fbee7: mov%gs:0x28,%rax
0.00 : 813fbef0: mov%rax,0xb0(%rsp)
0.00 : 813fbef8: xor%eax,%eax
: DEFINE_AUDIT_SK(sa, op, sk);
0.00 : 813fbefa: lea0x78(%rsp),%rdx
1.71 : 813fbeff: lea0x20(%rsp),%r8
0.00 : 813fbf04: movq $0x0,(%rsp)
0.00 : 813fbf0c: movq $0x0,0x10(%rsp)
0.00 : 813fbf15: mov%rdx,%rdi
14.53 : 813fbf18: rep stos %rax,%es:(%rdi)
1.71 : 813fbf1b: mov$0xb,%ecx
0.00 : 813fbf20: mov%r8,%rdi
0.00 : 813fbf23: mov%r14,0x80(%rsp)
18.80 : 813fbf2b: rep stos %rax,%es:(%rdi)
0.00 : 813fbf2e: mov%rsi,0x28(%rsp)
1.71 : 813fbf33: mov%r9w,0x88(%rsp)
0.00 : 813fbf3c: cmp$0x1,%r9w
0.00 : 813fbf41: je 813fbfa1
0.00 : 813fbf43: mov$0x2,%eax
0.00 : 813fbf48: test %r14,%r14
0.00 : 813fbf4b: je 813fbfa1
14.53 : 813fbf4d: mov%al,(%rsp)
0.00 : 813fbf50: movzwl 0x1ea(%r14),%eax
: AA_BUG(!sk);
:
: if (unconfined(label))
: return 0;
:
: return fn_for_each_confined(label,
profile,
0.00 : 813fbf58: xor%r12d,%r12d
: DEFINE_AUDIT_SK(sa, op, sk);
0.00 : 813fbf5b: mov%r8,0x18(%rsp)
8.55 : 813fbf60: mov%eax,0x58(%rsp)
0.00 : 813fbf64: movzbl 0x1e9(%r14),%eax
0.00 : 813fbf6c: mov%rdx,0x8(%rsp)
0.00 : 813fbf71: mov%eax,0x5c(%rsp)
: if (unconfined(label))
8.55 : 813fbf75: testb $0x2,0x40(%rbp)
0.00 : 813fbf79: je 813fbfa8
:
aa_profile_af_sk_perm(profile, , request, sk));
: }
0.00 : 813fbf7b: mov0xb0(%rsp),%rdx
Re: [PATCH] apparmor: Fix network performance issue in aa_label_sk_perm
On 09/07/2018 09:37 AM, John Johansen wrote:
> hey Tony,
>
> thanks for the patch, I am curious did you're investigation look
> into what parts of DEFINE_AUDIT_SK are causing the issue?
Hi JJ.
Attached are the perf annotations for DEFINE_AUDIT_SK (percentages are relative
to the fn).
Our kernel performance testing is carried out with default installs which means
AppArmor
is enabled but the performance tests are unconfined. It was obvious that the
overhead of
DEFINE_AUDIT_SK was significant for smaller packet sizes (typical of synthetic
benchmarks)
and that it didn't need to execute for the unconfined case, hence the patch.
I didn't
spend any time looking at the performance of confined tasks. It may be worth
your time to
look at this.
Comparing my current tip (2601dd392dd1) to tip+patch I'm seeing an increase of
3-6% in netperf
throughput for packet sizes 64-1024.
HTH
Tony
Percent | Source code & Disassembly of vmlinux for cycles:ppp (117
samples)
-
:
:
:
: Disassembly of section .text:
:
: 813fbec0 :
: aa_label_sk_perm():
:
type));
: }
:
: static int aa_label_sk_perm(struct aa_label
*label, const char *op, u32 request,
: struct sock *sk)
: {
0.00 : 813fbec0: callq 81a017f0 <__fentry__>
2.56 : 813fbec5: push %r14
0.00 : 813fbec7: mov%rcx,%r14
: struct aa_profile *profile;
: DEFINE_AUDIT_SK(sa, op, sk);
0.00 : 813fbeca: mov$0x7,%ecx
: {
0.00 : 813fbecf: push %r13
3.42 : 813fbed1: mov%edx,%r13d
0.00 : 813fbed4: push %r12
0.00 : 813fbed6: push %rbp
0.00 : 813fbed7: mov%rdi,%rbp
5.13 : 813fbeda: push %rbx
0.00 : 813fbedb: sub$0xb8,%rsp
: DEFINE_AUDIT_SK(sa, op, sk);
0.00 : 813fbee2: movzwl 0x10(%r14),%r9d
: {
1.71 : 813fbee7: mov%gs:0x28,%rax
0.00 : 813fbef0: mov%rax,0xb0(%rsp)
0.00 : 813fbef8: xor%eax,%eax
: DEFINE_AUDIT_SK(sa, op, sk);
0.00 : 813fbefa: lea0x78(%rsp),%rdx
1.71 : 813fbeff: lea0x20(%rsp),%r8
0.00 : 813fbf04: movq $0x0,(%rsp)
0.00 : 813fbf0c: movq $0x0,0x10(%rsp)
0.00 : 813fbf15: mov%rdx,%rdi
14.53 : 813fbf18: rep stos %rax,%es:(%rdi)
1.71 : 813fbf1b: mov$0xb,%ecx
0.00 : 813fbf20: mov%r8,%rdi
0.00 : 813fbf23: mov%r14,0x80(%rsp)
18.80 : 813fbf2b: rep stos %rax,%es:(%rdi)
0.00 : 813fbf2e: mov%rsi,0x28(%rsp)
1.71 : 813fbf33: mov%r9w,0x88(%rsp)
0.00 : 813fbf3c: cmp$0x1,%r9w
0.00 : 813fbf41: je 813fbfa1
0.00 : 813fbf43: mov$0x2,%eax
0.00 : 813fbf48: test %r14,%r14
0.00 : 813fbf4b: je 813fbfa1
14.53 : 813fbf4d: mov%al,(%rsp)
0.00 : 813fbf50: movzwl 0x1ea(%r14),%eax
: AA_BUG(!sk);
:
: if (unconfined(label))
: return 0;
:
: return fn_for_each_confined(label,
profile,
0.00 : 813fbf58: xor%r12d,%r12d
: DEFINE_AUDIT_SK(sa, op, sk);
0.00 : 813fbf5b: mov%r8,0x18(%rsp)
8.55 : 813fbf60: mov%eax,0x58(%rsp)
0.00 : 813fbf64: movzbl 0x1e9(%r14),%eax
0.00 : 813fbf6c: mov%rdx,0x8(%rsp)
0.00 : 813fbf71: mov%eax,0x5c(%rsp)
: if (unconfined(label))
8.55 : 813fbf75: testb $0x2,0x40(%rbp)
0.00 : 813fbf79: je 813fbfa8
:
aa_profile_af_sk_perm(profile, , request, sk));
: }
0.00 : 813fbf7b: mov0xb0(%rsp),%rdx
[PATCH] apparmor: Fix network performance issue in aa_label_sk_perm
The netperf benchmark shows a 5.73% reduction in throughput for
small (64 byte) transfers by unconfined tasks.
DEFINE_AUDIT_SK() in aa_label_sk_perm() should not be performed
unconditionally, rather only when the label is confined.
netperf-tcp
56974a6fc^ 56974a6fc
Min 64 563.48 ( 0.00%) 531.17 ( -5.73%)
Min 128 1056.92 ( 0.00%) 999.44 ( -5.44%)
Min 256 1945.95 ( 0.00%) 1867.97 ( -4.01%)
Min 1024 6761.40 ( 0.00%) 6364.23 ( -5.87%)
Min 2048 0.53 ( 0.00%)10606.20 ( -4.54%)
Min 3312 13692.67 ( 0.00%)13158.41 ( -3.90%)
Min 4096 14926.29 ( 0.00%)14457.46 ( -3.14%)
Min 8192 18399.34 ( 0.00%)18091.65 ( -1.67%)
Min 1638421384.13 ( 0.00%)21158.05 ( -1.06%)
Hmean 64 564.96 ( 0.00%) 534.38 ( -5.41%)
Hmean 128 1064.42 ( 0.00%) 1010.12 ( -5.10%)
Hmean 256 1965.85 ( 0.00%) 1879.16 ( -4.41%)
Hmean 1024 6839.77 ( 0.00%) 6478.70 ( -5.28%)
Hmean 2048 11154.80 ( 0.00%)10671.13 ( -4.34%)
Hmean 3312 13838.12 ( 0.00%)13249.01 ( -4.26%)
Hmean 4096 15009.99 ( 0.00%)14561.36 ( -2.99%)
Hmean 8192 18975.57 ( 0.00%)18326.54 ( -3.42%)
Hmean 1638421440.44 ( 0.00%)21324.59 ( -0.54%)
Stddev64 1.24 ( 0.00%)2.85 (-130.64%)
Stddev128 4.51 ( 0.00%)6.53 ( -44.84%)
Stddev256 11.67 ( 0.00%)8.50 ( 27.16%)
Stddev102448.33 ( 0.00%) 75.07 ( -55.34%)
Stddev204854.82 ( 0.00%) 65.16 ( -18.86%)
Stddev3312 153.57 ( 0.00%) 56.29 ( 63.35%)
Stddev4096 100.25 ( 0.00%) 88.50 ( 11.72%)
Stddev8192 358.13 ( 0.00%) 169.99 ( 52.54%)
Stddev16384 43.99 ( 0.00%) 141.82 (-222.39%)
Signed-off-by: Tony Jones
Fixes: 56974a6fcfef ("apparmor: add base infastructure for socket
mediation")
---
security/apparmor/net.c | 15 +--
1 file changed, 9 insertions(+), 6 deletions(-)
diff --git a/security/apparmor/net.c b/security/apparmor/net.c
index bb24cfa0a164..d5d72dd1ca1f 100644
--- a/security/apparmor/net.c
+++ b/security/apparmor/net.c
@@ -146,17 +146,20 @@ int aa_af_perm(struct aa_label *label, const char *op,
u32 request, u16 family,
static int aa_label_sk_perm(struct aa_label *label, const char *op, u32
request,
struct sock *sk)
{
- struct aa_profile *profile;
- DEFINE_AUDIT_SK(sa, op, sk);
+ int error = 0;
AA_BUG(!label);
AA_BUG(!sk);
- if (unconfined(label))
- return 0;
+ if (!unconfined(label)) {
+ struct aa_profile *profile;
+ DEFINE_AUDIT_SK(sa, op, sk);
- return fn_for_each_confined(label, profile,
- aa_profile_af_sk_perm(profile, , request, sk));
+ error = fn_for_each_confined(label, profile,
+ aa_profile_af_sk_perm(profile, , request, sk));
+ }
+
+ return error;
}
int aa_sk_perm(const char *op, u32 request, struct sock *sk)
--
2.18.0
