Michael Brown created IMPALA-6640: ------------------------------------- Summary: profile collection via standard Impyla usage won't have ExecSummary Key: IMPALA-6640 URL: https://issues.apache.org/jira/browse/IMPALA-6640 Project: IMPALA Issue Type: Bug Components: Infrastructure Affects Versions: Impala 2.11.0 Reporter: Michael Brown
It seems the Impyla (HS2 client for Python stress test) DB API paradigm doesn't lend itself to theĀ ideal collection of query profiles. # It seems a profile will not have an ExecSummary, End Time, or Errors unlessĀ the profile is requested after a CloseOperation RPC. This makes sense to me. # GetRuntimeProfile RPC must be called before the CloseSession RPC. This makes sense to me. # One can't really use standard, public methods in Impyla to do this: impala.hiveserver2.HiveServer2Cursor().close() handles both the CloseOperation and CloseSession RPCs in a single call. # The same is true of the cursor in a context manager \_\_exit\_\_ setting. \_\_exit\_\_ just calls close() This means we have to tap into the more private bits of Impyla to do this. {noformat} #!/usr/bin/env impala-python import impala.dbapi from impala._thrift_api import TGetRuntimeProfileReq with impala.dbapi.connect(host='localhost', port=21050) as conn: with conn.cursor() as cursor: cursor.execute('SELECT COUNT(2) FROM functional.alltypes') cursor.fetchall() with open('profile-before-close-operation.txt', 'w') as fh: fh.write(cursor.get_profile()) # Normal cursor context would end here, but.... # Issue CloseOperation RPC without CloseSession RPC. # This is impala.hiveserver2.Operation().close() resp = cursor._last_operation.close() req = TGetRuntimeProfileReq(operationHandle=cursor._last_operation.handle, sessionHandle=cursor._last_operation.session.handle) resp = cursor._last_operation._rpc('GetRuntimeProfile', req) # impala.hiveserver2.HiveServer2Cursor()._reset_state() assist cursor._last_operation_active = False with open('profile-after-close-operation.txt', 'w') as fh: fh.write(resp.profile) {noformat} We should think about pythonic ways to make this better for Impyla, or at least alter the stress test to get the profiles of the queries after CloseOperation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)