[
https://issues.apache.org/jira/browse/IMPALA-13729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17923575#comment-17923575
]
Quanlong Huang commented on IMPALA-13729:
-----------------------------------------
Some logs of the KILL statement:
{noformat}
I0201 22:56:47.857080 29567 impala-server.cc:1416] Found local timezone
"America/Los_Angeles".
I0201 22:56:47.857198 29567 impala-server.cc:1475]
f840d9f30501a6a3:3a2ccae100000000] Registered query
query_id=f840d9f30501a6a3:3a2ccae100000000
session_id=5147eb358ff2df83:cd8f44e372870a97
I0201 22:56:47.863487 30622 control-service.cc:148]
9941dc9c3c2257bb:0e610b3900000000] ExecQueryFInstances():
query_id=9941dc9c3c2257bb:0e610b3900000000
coord=impala-ec2-centos79-m6i-4xlarge-xldisk-0853.vpc.cloudera.com:27000
#instances=1
I0201 22:56:47.863961 29564 coordinator.cc:590]
9941dc9c3c2257bb:0e610b3900000000] started execution on 1 backends for
query_id=9941dc9c3c2257bb:0e610b3900000000
I0201 22:56:47.869813 29567 Frontend.java:2190]
f840d9f30501a6a3:3a2ccae100000000] Analyzing query: KILL QUERY
'9941dc9c3c2257bb:0e610b3900000000' db: default
I0201 22:56:47.869855 29567 Frontend.java:2202]
f840d9f30501a6a3:3a2ccae100000000] The original executor group sets from
executor membership snapshot: [TExecutorGroupSet(curr_num_executors:3,
expected_num_executors:20, exec_group_name_prefix:)]
I0201 22:56:47.869887 29567 RequestPoolService.java:198]
f840d9f30501a6a3:3a2ccae100000000] Default pool only, scheduler allocation is
not specified.
I0201 22:56:47.869917 29567 Frontend.java:2223]
f840d9f30501a6a3:3a2ccae100000000] A total of 1 executor group sets to be
considered for auto-scaling: [TExecutorGroupSet(curr_num_executors:3,
expected_num_executors:20, exec_group_name_prefix:,
max_mem_limit:9223372036854775807, num_cores_per_executor:2147483647)]
I0201 22:56:47.869946 29567 Frontend.java:2264]
f840d9f30501a6a3:3a2ccae100000000] Consider executor group set:
TExecutorGroupSet(curr_num_executors:3, expected_num_executors:20,
exec_group_name_prefix:, max_mem_limit:9223372036854775807,
num_cores_per_executor:2147483647) with assumption of 0 cores per node.
I0201 22:56:47.870134 29567 AnalysisContext.java:530]
f840d9f30501a6a3:3a2ccae100000000] Analysis took 0 ms
I0201 22:56:47.870175 29567 BaseAuthorizationChecker.java:114]
f840d9f30501a6a3:3a2ccae100000000] Authorization check took 0 ms
I0201 22:56:47.870198 29567 Frontend.java:2600]
f840d9f30501a6a3:3a2ccae100000000] Analysis and authorization finished.
I0201 22:56:47.870267 29567 Frontend.java:2481]
f840d9f30501a6a3:3a2ccae100000000] Selected executor group:
TExecutorGroupSet(curr_num_executors:3, expected_num_executors:20,
exec_group_name_prefix:, max_mem_limit:9223372036854775807,
num_cores_per_executor:2147483647), reason: query is not auto-scalable
I0201 22:56:47.870607 29567 client-request-state.cc:2553]
f840d9f30501a6a3:3a2ccae100000000] Exec KillQuery:
query_id=9941dc9c3c2257bb:0e610b3900000000, requesting_user=jenkins, is_admin=1
I0201 22:56:47.870618 29567 impala-server.cc:1826]
f840d9f30501a6a3:3a2ccae100000000] KillQuery():
query_id=9941dc9c3c2257bb:0e610b3900000000, requesting_user=jenkins, is_admin=1
I0201 22:56:47.870627 29567 impala-server.cc:1813]
f840d9f30501a6a3:3a2ccae100000000] Cancel():
query_id=9941dc9c3c2257bb:0e610b3900000000
I0201 22:56:47.870637 29567 impala-server.cc:1580]
f840d9f30501a6a3:3a2ccae100000000] UnregisterQuery():
query_id=9941dc9c3c2257bb:0e610b3900000000
I0201 22:56:47.872282 29569 query-state.cc:1006]
9941dc9c3c2257bb:0e610b3900000000] Executing instance.
instance_id=9941dc9c3c2257bb:0e610b3900000000 fragment_idx=0
per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=5
I0201 22:56:47.872435 29564 client-request-state.cc:680]
9941dc9c3c2257bb:0e610b3900000000] Cancelled right after starting the
coordinator query id=9941dc9c3c2257bb:0e610b3900000000
I0201 22:56:47.872465 29564 coordinator.cc:796]
9941dc9c3c2257bb:0e610b3900000000] ExecState: query
id=9941dc9c3c2257bb:0e610b3900000000 execution cancelled
I0201 22:56:47.872478 29564 coordinator-backend-state.cc:1026]
9941dc9c3c2257bb:0e610b3900000000] query_id=9941dc9c3c2257bb:0e610b3900000000
target backend=127.0.0.1:27000: Sending CancelQueryFInstances rpcI0201
22:56:47.872594 30622 control-service.cc:239] CancelQueryFInstances():
query_id=9941dc9c3c2257bb:0e610b3900000000
I0201 22:56:47.872601 30622 query-exec-mgr.cc:132] QueryState:
query_id=9941dc9c3c2257bb:0e610b3900000000 refcnt=4
I0201 22:56:47.872604 30622 query-state.cc:1032] Cancel:
query_id=9941dc9c3c2257bb:0e610b3900000000
I0201 22:56:47.872609 30622 krpc-data-stream-mgr.cc:339] cancelling active
streams for query_id=9941dc9c3c2257bb:0e610b3900000000
I0201 22:56:47.872668 29564 coordinator.cc:1087]
9941dc9c3c2257bb:0e610b3900000000] CancelBackends()
query_id=9941dc9c3c2257bb:0e610b3900000000, tried to cancel 1 backends
I0201 22:56:47.872689 29564 coordinator.cc:1465]
9941dc9c3c2257bb:0e610b3900000000] Release admission control resources for
query_id=9941dc9c3c2257bb:0e610b3900000000
I0201 22:56:47.873696 31076 impala-server.cc:1617] Query successfully
unregistered: query_id=9941dc9c3c2257bb:0e610b3900000000
I0201 22:56:47.873981 29563 impala-beeswax-server.cc:242]
get_results_metadata(): query_id=9941dc9c3c2257bb:0e610b3900000000
I0201 22:56:47.873998 29563 impala-server.cc:1770]
9941dc9c3c2257bb:0e610b3900000000] Invalid or unknown query handle:
9941dc9c3c2257bb:0e610b3900000000. {noformat}
> TestKillQuery.test_same_coordinator fails in error_msg_expected
> ---------------------------------------------------------------
>
> Key: IMPALA-13729
> URL: https://issues.apache.org/jira/browse/IMPALA-13729
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Xuebin Su
> Priority: Critical
>
> Saw this failed in a downstream build.
> Stacktrace
> {noformat}
> query_test/test_kill_query.py:38: in test_same_coordinator
> assert_kill_ok(client, query_id_to_kill)
> util/cancel_util.py:65: in __exit__
> self.client.get_query_id(self.handle),
> E assert (error_msg_expected('TException - service has thrown:
> BeeswaxException(message=Query 9941dc9c3c2257bb:0e610b3900000000
> failed:\nInvalid or...1dc9c3c2257bb:0e610b3900000000.\n\n, log_context=,
> handle=QueryHandle(id=, log_context=), errorCode=0, SQLState=HY000)',
> 'Invalid or unknown query handle', '9941dc9c3c2257bb:0e610b3900000000') or
> error_msg_expected('TException - service has thrown:
> BeeswaxException(message=Query 9941dc9c3c2257bb:0e610b3900000000
> failed:\nInvalid or...1dc9c3c2257bb:0e610b3900000000.\n\n, log_context=,
> handle=QueryHandle(id=, log_context=), errorCode=0, SQLState=HY000)',
> 'Cancelled', '9941dc9c3c2257bb:0e610b3900000000'))
> E + where 'TException - service has thrown:
> BeeswaxException(message=Query 9941dc9c3c2257bb:0e610b3900000000
> failed:\nInvalid or...1dc9c3c2257bb:0e610b3900000000.\n\n, log_context=,
> handle=QueryHandle(id=, log_context=), errorCode=0, SQLState=HY000)' =
> str(ImpalaBeeswaxException())
> E + where ImpalaBeeswaxException() = ImpalaBeeswaxException()
> E + where ImpalaBeeswaxException() =
> <tests.util.cancel_util.QueryToKill instance at 0x7ff088362560>.exc
> E + and '9941dc9c3c2257bb:0e610b3900000000' = <bound method
> BeeswaxConnection.get_query_id of
> <tests.common.impala_connection.BeeswaxConnection object at
> 0x7ff087f4fb10>>(<tests.common.impala_connection.OperationHandle object at
> 0x7ff087f4f050>)
> E + where <bound method BeeswaxConnection.get_query_id of
> <tests.common.impala_connection.BeeswaxConnection object at 0x7ff087f4fb10>>
> = <tests.common.impala_connection.BeeswaxConnection object at
> 0x7ff087f4fb10>.get_query_id
> E + where <tests.common.impala_connection.BeeswaxConnection object at
> 0x7ff087f4fb10> = <tests.common.impala_connection.BeeswaxConnection object at
> 0x7ff087f4fb10>
> E + where <tests.common.impala_connection.BeeswaxConnection object
> at 0x7ff087f4fb10> = <tests.util.cancel_util.QueryToKill instance at
> 0x7ff088362560>.client
> E + and <tests.common.impala_connection.OperationHandle object at
> 0x7ff087f4f050> = <tests.common.impala_connection.OperationHandle object at
> 0x7ff087f4f050>
> E + where <tests.common.impala_connection.OperationHandle object at
> 0x7ff087f4f050> = <tests.util.cancel_util.QueryToKill instance at
> 0x7ff088362560>.handle
> E + and 'TException - service has thrown:
> BeeswaxException(message=Query 9941dc9c3c2257bb:0e610b3900000000
> failed:\nInvalid or...1dc9c3c2257bb:0e610b3900000000.\n\n, log_context=,
> handle=QueryHandle(id=, log_context=), errorCode=0, SQLState=HY000)' =
> str(ImpalaBeeswaxException())
> E + where ImpalaBeeswaxException() = ImpalaBeeswaxException()
> E + where ImpalaBeeswaxException() =
> <tests.util.cancel_util.QueryToKill instance at 0x7ff088362560>.exc
> E + and '9941dc9c3c2257bb:0e610b3900000000' = <bound method
> BeeswaxConnection.get_query_id of
> <tests.common.impala_connection.BeeswaxConnection object at
> 0x7ff087f4fb10>>(<tests.common.impala_connection.OperationHandle object at
> 0x7ff087f4f050>)
> E + where <bound method BeeswaxConnection.get_query_id of
> <tests.common.impala_connection.BeeswaxConnection object at 0x7ff087f4fb10>>
> = <tests.common.impala_connection.BeeswaxConnection object at
> 0x7ff087f4fb10>.get_query_id
> E + where <tests.common.impala_connection.BeeswaxConnection object at
> 0x7ff087f4fb10> = <tests.common.impala_connection.BeeswaxConnection object at
> 0x7ff087f4fb10>
> E + where <tests.common.impala_connection.BeeswaxConnection object
> at 0x7ff087f4fb10> = <tests.util.cancel_util.QueryToKill instance at
> 0x7ff088362560>.client
> E + and <tests.common.impala_connection.OperationHandle object at
> 0x7ff087f4f050> = <tests.common.impala_connection.OperationHandle object at
> 0x7ff087f4f050>
> E + where <tests.common.impala_connection.OperationHandle object at
> 0x7ff087f4f050> = <tests.util.cancel_util.QueryToKill instance at
> 0x7ff088362560>.handle{noformat}
> Standard Error
> {noformat}
> SET
> client_identifier=query_test/test_kill_query.py::TestKillQuery::()::test_same_coordinator[protocol:beeswax];
> -- connecting to: localhost:21000
> -- 2025-02-01 22:56:47,787 INFO MainThread: Could not connect to ('::1',
> 21000, 0, 0)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-core-ozone/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-core-ozone/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- connecting to localhost:21050 with impyla
> -- 2025-02-01 22:56:47,788 INFO MainThread: Could not connect to ('::1',
> 21050, 0, 0)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-core-ozone/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-core-ozone/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- 2025-02-01 22:56:47,807 INFO MainThread: Closing active operation
> -- connecting to localhost:28000 with impyla
> -- 2025-02-01 22:56:47,835 INFO MainThread: Closing active operation
> -- connecting to localhost:11050 with impyla
> -- 2025-02-01 22:56:47,837 INFO MainThread: Could not connect to ('::1',
> 11050, 0, 0)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-core-ozone/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-core-ozone/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- 2025-02-01 22:56:47,837 INFO MainThread: Could not connect to
> ('127.0.0.1', 11050)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-core-ozone/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-core-ozone/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- 2025-02-01 22:56:47,837 ERROR MainThread: Could not connect to any of
> [('::1', 11050, 0, 0), ('127.0.0.1', 11050)]
> -- 2025-02-01 22:56:47,837 INFO MainThread: HS2 FENG connection setup
> failed, continuing...: Could not connect to any of [('::1', 11050, 0, 0),
> ('127.0.0.1', 11050)]
> SET
> client_identifier=query_test/test_kill_query.py::TestKillQuery::()::test_same_coordinator[protocol:beeswax];
> -- connecting to: localhost:21000
> -- 2025-02-01 22:56:47,839 INFO MainThread: Could not connect to ('::1',
> 21000, 0, 0)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-core-ozone/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-core-ozone/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> SET
> client_identifier=query_test/test_kill_query.py::TestKillQuery::()::test_same_coordinator[protocol:beeswax];
> -- connecting to: localhost:21000
> -- 2025-02-01 22:56:47,839 INFO MainThread: Could not connect to ('::1',
> 21000, 0, 0)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-core-ozone/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-core-ozone/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- executing async: localhost:21000
> SELECT sleep(1000);
> -- 2025-02-01 22:56:47,855 INFO MainThread: Started query
> 9941dc9c3c2257bb:0e610b3900000000
> -- 9941dc9c3c2257bb:0e610b3900000000: fetching all rows
> -- executing against localhost:21000
> KILL QUERY '9941dc9c3c2257bb:0e610b3900000000';
> -- 2025-02-01 22:56:47,873 INFO MainThread: Started query
> f840d9f30501a6a3:3a2ccae100000000
> -- closing connection to: localhost:21000{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
