Nghia Le commented on IMPALA-7259:

I have done some experiments using cProfile with your file and found out that 
there are 2 main functions which made the shell slow with large queries: 
parse_query_text() and strip_leading_comment() in impala_shell.py

For parse_query_text():

    The problem is in this statement


    ``` if query_list and not sqlparse.format(query_list[-1], 



    Since your last query is a really long string, the sqlparse.format needs a 
lot of time to check whether that string is a comment or not.


For strip_leading_comment():

    The problem also lies at string processing part.

> impala-shell is weirdly slow with some large queries
> ----------------------------------------------------
>                 Key: IMPALA-7259
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7259
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Clients
>    Affects Versions: Impala 3.1.0
>            Reporter: Tim Armstrong
>            Assignee: Nghia Le
>            Priority: Major
>         Attachments: wide-parquet-agg.sql
> impala-shell is very slow at processing some large queries - it takes over a 
> minute to actually submit the query. I've attached an example.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

Reply via email to