Michael Brown created IMPALA-6640:
Summary: profile collection via standard Impyla usage won't have
Issue Type: Bug
Affects Versions: Impala 2.11.0
Reporter: Michael Brown
It seems the Impyla (HS2 client for Python stress test) DB API paradigm doesn't
lend itself to the ideal collection of query profiles.
# It seems a profile will not have an ExecSummary, End Time, or Errors unless
the profile is requested after a CloseOperation RPC. This makes sense to me.
# GetRuntimeProfile RPC must be called before the CloseSession RPC. This makes
sense to me.
# One can't really use standard, public methods in Impyla to do this:
impala.hiveserver2.HiveServer2Cursor().close() handles both the CloseOperation
and CloseSession RPCs in a single call.
# The same is true of the cursor in a context manager \_\_exit\_\_ setting.
\_\_exit\_\_ just calls close()
This means we have to tap into the more private bits of Impyla to do this.
from impala._thrift_api import TGetRuntimeProfileReq
with impala.dbapi.connect(host='localhost', port=21050) as conn:
with conn.cursor() as cursor:
cursor.execute('SELECT COUNT(2) FROM functional.alltypes')
with open('profile-before-close-operation.txt', 'w') as fh:
# Normal cursor context would end here, but....
# Issue CloseOperation RPC without CloseSession RPC.
# This is impala.hiveserver2.Operation().close()
resp = cursor._last_operation.close()
req = TGetRuntimeProfileReq(operationHandle=cursor._last_operation.handle,
resp = cursor._last_operation._rpc('GetRuntimeProfile', req)
# impala.hiveserver2.HiveServer2Cursor()._reset_state() assist
cursor._last_operation_active = False
with open('profile-after-close-operation.txt', 'w') as fh:
We should think about pythonic ways to make this better for Impyla, or at least
alter the stress test to get the profiles of the queries after CloseOperation.
This message was sent by Atlassian JIRA