Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/8611 )
Change subject: IMPALA-6225: Query profile date-time strings should have ns precision. ...................................................................... Patch Set 6: (2 comments) > query_profile_encoded gives the raw thrift object. Tools that use the profile tend to use this. To use this, I've included the magic incantation below. Profiles are zlib compressed, base64 encoded Thrift serializations, and can be extracted. I think it'd be good for us to have the incantation checked into a test. It's definitely pretty opaque. $impala-python test.py Query (id=324ec7e76a7796c0:eb4da30800000000) $cat test.py from thrift.protocol import TCompactProtocol from thrift.TSerialization import deserialize from RuntimeProfile.ttypes import TRuntimeProfileTree import base64 import zlib b = "eJyVVN9v21QUdsfEQlq6VV1HooJ0QUi0Umvd6x+xE1FpIXaJRdp0cdpp0qTqxj5urnDs9Pq6U3lCaA/T/g6eEPwBiHceeeaRP4MHHriuaZdWC+r85HPs8+M75ztf/cva1pMc+DnaYOGO0TBAt6gNkREQ3Wy2wNaamoGvns21u/Wt2oqX0ECwMximgsZDNoG16iNFqX1w3X60sLKu1Cvr8ss9P59MKD9fU94tfPXVcidNEggEhOgwA742HbOYTR93ev1Dxx201V6/0+4tdtKUhyyhIuWfjSjELBtvY6yrEI7VIE7zEDhVg3TS0jSJouI4PTQ8n8J7h75bdSCieSyQM7oXlq/LDsRwQi9rKhU3CVHR1ycaJtY2IdsaQcRs6bhl2CppNIymKdN+3JWYfOBnwDV0wFORBmmMjoBnLE3uHDWWvcmUxvTS84xdmCE6K22kqQSreDsIx6ZKdBWjgdtz276LNkY5i0OkmYTY2IwIUNpskjAiRgNrViOwoBmRZmiTZhAaYGze3wfxIuXfonYYcsiyVWLJ3JZKbF3VcMvEMg6XW+9PhSydoY0MBBqdoyBNInaSc1q4EZW4pzGVC+CbuPuV7OW40+50vf2vdwjGW3vu3nHP2/OGOxoxLMPWG4YtvcNjp3+wgz+/RYXNL26ZdbFM5gu5lcqut+/5XddZeuPMszv9b6qlfbFZueKqL7EXMDznU0tiNi3Qo5FmmUDNlhzWyNDCAEdYD2yqLV3+XEQvdr0j13cHR+5A+9A/jcu6E0jEkmQMyo9LFlalm4v5zMCmbZPibO7Op27979WZPq918b98m ql9Rc8Z/LPzujamGyS86OzGjd2g/002zRzM9dnMXuGtlv/OHLw63PVCR5T6+/WKUlv+D7UcQMwSqO/+8f3Ky19/WPjrt1+qP/7zZ1Tv3i//yPLRhAkJa+WgyMeSExSxREoFhA8GcJpDJt44DhMOJywTwNFpES31aakcXbkRqWS92sNOzCTwXRDB+CllouiBS/H6/eefnitzde6jQfpiT86MMxqz7y5QXgbOF9GF18rGW8vNy6Yo/wKFx9Ux" b = "eJztWFtsI1cZHjvZbHCyubDrMO6u4NBtUSzF3hnHt1gEcOLJxtrEcW1ntwhQdDxz7Ix2Lt6ZcbKp+lDxQJcFClSo6gMSRdpWWmlV7UsrUSSEoKCqgmpfkIoEdLmogMQTTxS1iP/MjO3xrUkeQEjsRJHnnP+c//yX77+cCf2GXXisSYwDNC9Ly4uxOBFTJJXEqdRSUuQypBqX8CKX5twnHBwLLbCzeQ2LlrxHKrqFlYqskmBgjmHYj3SP53yzZ303PpoTVrYvos2tnICuZEuFfOGi6pzYMPSarBAkGgRbREL7u3RkNDVN1uoII2djtSkrEtJrKK82sIKjaNskqCRsCNmy4BBNZOlIJdhsGgRdc1gTo6YbKtZEEg2xA0Q4C+KeLDdVFRsHQSb0afYTq7raaFqkLGKthLU6yZqmXNdUollUIcPR8BiaszdmspIqAxddQwYxm4p1hk5YVFVZVYkkg9bKwdSqrmlEpLOgmTHWACvIjYlVXTckWcOWbpxqYDAJMTKxGLggkCM1DMxQrnpScl6nckQhddxiwYwLmoSoMA/FOD4V4flIjEd8IhNPZeKxaDqdiCfTY4Jh6IbJzAmmJav23iIxIuu6aaFNoo7zXDydSCW5CeE6EV1Dfdsf2ALLUpkQQufpWhOh7F7dPgyhTXzdfTtf0veBBLyjrfciwVcpZ3e2M8wRC8sKCkT+Y0+A4zLbhfxWAbkPD//xaCqxqJpdby7Ju8x5ONT1dIZTDirRZWJQR1+W7aGE9pwxikV5PspFyoVssby+VXExPe+AOp6I4zgGH8ZjyVQsgavpNJcUa2Kaj9eqabIk1RZhjiTC0wVi7evGVZSVJICSOZfJ1ODJ8LFUlIM/HlzLx5MseBA5HpQ1VCKAhT1sgRizXQia58JotKhg7eWRPqMHqA/bQAAWetMQiZdXhnpNNw6WuZVA/8IWmsz2Mh4E5DZXAqu6ROpEQ5Js4qoCcKtCmIIQGjECgTXwUHEjW0BrpezFTaFQQV/YLhSzpUq+Am4Tcl9CuxRsyzySNdOiYQ3vgSdRv6hwsErUCHHlACntsdFRwBYczoqUtsAh5XzhEmV0hE1PtnEUAM+Lui2JFdFpSGgSFQgdiQ+sspoNhURk2MQhQ9+PmPITZJlfQSK2g16RLTBcn3M4J29uNSgnE82bxKJWBElqcr1p2AcgEKVl1/DnCtubOwVIe+VlfoG+l1ezhYJQ2qmsl4Rsjs6WtguV/Kaws5bfqACB5shlbmGzspPbKi5zjxzhxDAz4awqW6Dz+Fq+kC+vC7nJzmTT9G9dCjjjykGDnHhsWyh9frJEIFmD64q6rky6qSzSgEGgTJy0mc99cikVJxzHp5M4mahyKZKJx0VxUUwmSZXnRFzlJluLKeeTK4JQvpJ9/FT5muLIQxP4uAn5UbTQYgCmDGt4aozHkzwXG/Xk4dA/WI80XWd5mLUTrkdJr1G6bNGTMuzTeopAT0LvDX9PCehW1Fs1juS6YyPKzhtDasbQ9NPl6b6q6K0wblU667vJHFqRoRGZKTpi2bZXZI2ELn/37j/xG3e+9mXfW7/42fd9//rz32/7vvPi0/d9ocdnsxBYB6ZsopqsyeYukc4J15ryHlYI5BMkKtg0iQm626dKbBm6EOhHNMhbtvqtPoXNyaZlyFW6qIswa0tDe5fWAQw75QKiJd9z7z5/z/fmG9/7qf93z7z3lv8Pv3r2nv/+B7d+7X/pWx/81f/ai698c+Tduz94duS1n7z/wkjo+Wlnu9msOo1D/xFnyjYNQbuDcMuyp6npFEIFbM+dKxEsHdBWybRxC/bnURWLVwnkrlRWUWBIwBNN2+ctAprnUc3AdWr1Tu4NOzyINGXXdrwHBZwm9dk12QA3Q0ZDNWKJIN7MtmaQOhgMnGS3ZtAsTToBUAZ4QF/FhDbYM6uKDAes0T1XsNzuuP726jsnhjddH4PDN8HwhgwJ8wkbaa2Nf3rvnYmBXRkTGqfgmh944jCGDLsotC1TdLvWQ3vl4ChoNr0mK8DQLBGRgAZSkLHbyDU7yXfLfNT28s37T120dfHfCK04bnLc0WwgBaTXRJmYz/hW9aZmZRC/gFSIxgtIxddhlIA+54L9s4BiCWsXPRoBZTLuTILrnUn1rVnqW7PUv2YputQ99xDNDQUaSq3GD3oDSG64Ttju5oSPQb+ALq2EUWh+uH4fws/2b6/d+y3OsMEs4A82SGithXDoQ+CG8/4se9qlVXYhtqWKDlKYwUnqi5kVRddVh/3KAXQ6wbEPc900lc8Rz5YOVj/1Y58778mRDpfTdBZSvjSIYtDc6rLr4jVJoxBwKTVFG2F+iA1bDLduuEboCHXGSy2DgXtJZSggcEArMhzSpxzVbIuYeW1PVwBhkLShdlnkulXel2nImw7ET3sXlw/MDpuPeyn2+xWsKKuKLl51F937yp0TLUncdbQMdlg84iVdHi4H2xM6b//yh+ft0Bm9MSM6KZKGNBQz6BshTDKJxWgsGad3ARounhHB0Ph2xqYlSWQvQ1tbTjOnO3nToL3vZygrSkMXTCI6vLxDyswz9jCzZ05pTbWTa338hNmAhhDRHtF8mEZzBnELTjzDL96r278uF8SF1r0b+vTsFbb7tLP+m8zbg8DfB/teYPcCehCUB4G4C7zDYDsQsAOhekSQHgLCIwHsEB7zg0JgIKgZ9nw7A+VdT6ANuUbEAxEqDSyBig94Dn2dHaeFyEXzndtfhTCZsWc MQjyUn//xts9DgUu75lJefvVhiM0WoWiQBjZaieGFu98YGZ7Ixj1cfvTbRxl2onv360///tzQkjt6k0m3RW9z8nJoU/sVaq/v18jLYaBa0IHRu94Ovevt0Lue/bHnGMl68Mcdt43o2wDmsy+I9qXL/qTGheHELx7zRDsmSsRqGloroc94p0oQuEH/3K2RHuGeu/X6KGTftnzpXvZdjPtYMizruUB0V0X/8b77UQGA8tk2nA9tl9A8veIvdzUD4QfF+P+gGFOsPCg7//2yE15gJ7qCIfiXk6ExSOzOOmeScSYflKgHJep/okQx/wZYqp7y" tree = TRuntimeProfileTree() deserialize(tree, zlib.decompress(base64.b64decode(b)), protocol_factory=TCompactProtocol.TCompactProtocolFactory()) print tree.nodes[0].name http://gerrit.cloudera.org:8080/#/c/8611/6/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/8611/6/tests/query_test/test_observability.py@151 PS6, Line 151: print 'Test case will fail' I think the following will fail more obviously: assert False, "Debug profile for query not available in 120 seconds." http://gerrit.cloudera.org:8080/#/c/8611/6/tests/query_test/test_observability.py@152 PS6, Line 152: assert len(start_time_sub_sec_str) == 9 I don't think this line is ever reached, because in the successful case, line 141 has exited the test. -- To view, visit http://gerrit.cloudera.org:8080/8611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e124b9c7e0717b8dc2cdab46aea41d74c5f2fd0 Gerrit-Change-Number: 8611 Gerrit-PatchSet: 6 Gerrit-Owner: Zoram Thanga <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Philip Zeyliger <[email protected]> Gerrit-Reviewer: Zoram Thanga <[email protected]> Gerrit-Comment-Date: Tue, 21 Nov 2017 23:50:48 +0000 Gerrit-HasComments: Yes
