Michael Brown created IMPALA-6640:

             Summary: profile collection via standard Impyla usage won't have 
                 Key: IMPALA-6640
                 URL: https://issues.apache.org/jira/browse/IMPALA-6640
             Project: IMPALA
          Issue Type: Bug
          Components: Infrastructure
    Affects Versions: Impala 2.11.0
            Reporter: Michael Brown

It seems the Impyla (HS2 client for Python stress test) DB API paradigm doesn't 
lend itself to theĀ ideal collection of query profiles.
 # It seems a profile will not have an ExecSummary, End Time, or Errors unlessĀ 
the profile is requested after a CloseOperation RPC. This makes sense to me.
 # GetRuntimeProfile RPC must be called before the CloseSession RPC. This makes 
sense to me.
 # One can't really use standard, public methods in Impyla to do this: 
impala.hiveserver2.HiveServer2Cursor().close() handles both the CloseOperation 
and CloseSession RPCs in a single call.
 # The same is true of the cursor in a context manager \_\_exit\_\_ setting. 
\_\_exit\_\_ just calls close()

This means we have to tap into the more private bits of Impyla to do this.
#!/usr/bin/env impala-python

import impala.dbapi
from impala._thrift_api import TGetRuntimeProfileReq

with impala.dbapi.connect(host='localhost', port=21050) as conn:
  with conn.cursor() as cursor:
    cursor.execute('SELECT COUNT(2) FROM functional.alltypes')
    with open('profile-before-close-operation.txt', 'w') as fh:
    # Normal cursor context would end here, but....
    # Issue CloseOperation RPC without CloseSession RPC.
    # This is impala.hiveserver2.Operation().close()
    resp = cursor._last_operation.close()
    req = TGetRuntimeProfileReq(operationHandle=cursor._last_operation.handle,
    resp = cursor._last_operation._rpc('GetRuntimeProfile', req)
    # impala.hiveserver2.HiveServer2Cursor()._reset_state() assist
    cursor._last_operation_active = False
    with open('profile-after-close-operation.txt', 'w') as fh:

We should think about pythonic ways to make this better for Impyla, or at least 
alter the stress test to get the profiles of the queries after CloseOperation.

This message was sent by Atlassian JIRA

Reply via email to