[ https://issues.apache.org/jira/browse/IMPALA-13037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840619#comment-17840619 ]
Quanlong Huang commented on IMPALA-13037: ----------------------------------------- Also checked logs of hive-server2. There is an application id printed by the same thread: {noformat} 2024-04-22T20:17:56,360 INFO [HiveServer2-Background-Pool: Thread-159] tez.TezTask: Subscribed to counters: [] for queryId: jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df 2024-04-22T20:17:56,360 INFO [HiveServer2-Background-Pool: Thread-159] tez.TezTask: Tez session hasn't been created yet. Opening session 2024-04-22T20:17:56,360 INFO [HiveServer2-Background-Pool: Thread-159] tez.TezSessionState: User of session id d6d65f07-cdff-4f5c-bbb0-b2fa24d2d1cc is jenkins 2024-04-22T20:17:56,369 INFO [HiveServer2-Background-Pool: Thread-159] tez.DagUtils: Localizing resource because it does not exist: file:/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/fe/target/dependency/postgresql-42.5.1.jar to dest: hdfs://localhost:20500/tmp/hive/jenkins/_tez_session_dir/d6d65f07-cdff-4f5c-bbb0-b2fa24d2d1cc-resources/postgresql-42.5.1.jar 2024-04-22T20:17:56,549 INFO [HiveServer2-Background-Pool: Thread-159] tez.DagUtils: Resource modification time: 1713842276519 for hdfs://localhost:20500/tmp/hive/jenkins/_tez_session_dir/d6d65f07-cdff-4f5c-bbb0-b2fa24d2d1cc-resources/postgresql-42.5.1.jar 2024-04-22T20:17:56,625 INFO [HiveServer2-Background-Pool: Thread-159] tez.TezSessionState: Created new resources: null 2024-04-22T20:17:56,627 INFO [HiveServer2-Background-Pool: Thread-159] tez.DagUtils: Jar dir is null / directory doesn't exist. Choosing HIVE_INSTALL_DIR - /user/jenkins/.hiveJars 2024-04-22T20:17:57,105 INFO [HiveServer2-Background-Pool: Thread-159] tez.TezSessionState: Computed sha: 77f0dcaafc28cfe7b2d805cdf2d3a083370b2299011e98eb893bd9573e3d4c10 for file: file:/data0/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/cdp_components-45689292/apache-hive-3.1.3000.7.2.18.0-369-bin/lib/hive-exec-3.1.3000.7.2.18.0-369.jar of length: 74.73MB in 474 ms 2024-04-22T20:17:57,109 INFO [HiveServer2-Background-Pool: Thread-159] tez.DagUtils: Resource modification time: 1713837749334 for hdfs://localhost:20500/user/jenkins/.hiveJars/hive-exec-3.1.3000.7.2.18.0-369-77f0dcaafc28cfe7b2d805cdf2d3a083370b2299011e98eb893bd9573e3d4c10.jar 2024-04-22T20:17:57,227 INFO [HiveServer2-Background-Pool: Thread-159] counters.Limits: Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200 2024-04-22T20:17:57,227 INFO [HiveServer2-Background-Pool: Thread-159] counters.Limits: Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=120 2024-04-22T20:17:57,227 INFO [HiveServer2-Background-Pool: Thread-159] client.TezClient: Tez Client Version: [ component=tez-api, version=0.9.1.7.2.18.0-369, revision=590a68b8a743783155fea2e6f2026f01a8775635, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=2023-09-28T12:31:39Z ] 2024-04-22T20:17:57,227 INFO [HiveServer2-Background-Pool: Thread-159] tez.TezSessionState: Opening new Tez Session (id: d6d65f07-cdff-4f5c-bbb0-b2fa24d2d1cc, scratch dir: hdfs://localhost:20500/tmp/hive/jenkins/_tez_session_dir/d6d65f07-cdff-4f5c-bbb0-b2fa24d2d1cc) 2024-04-22T20:17:57,293 INFO [HiveServer2-Background-Pool: Thread-159] client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 2024-04-22T20:17:57,575 INFO [HiveServer2-Background-Pool: Thread-159] client.TezClient: Session mode. Starting session. 2024-04-22T20:17:57,664 INFO [HiveServer2-Background-Pool: Thread-159] client.TezClientUtils: Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true 2024-04-22T20:17:57,675 INFO [HiveServer2-Background-Pool: Thread-159] client.TezClient: Tez system stage directory hdfs://localhost:20500/tmp/hive/jenkins/_tez_session_dir/d6d65f07-cdff-4f5c-bbb0-b2fa24d2d1cc/.tez/application_1713840366821_0001 doesn't exist and is created 2024-04-22T20:17:57,699 INFO [HiveServer2-Background-Pool: Thread-159] conf.Configuration: resource-types.xml not found 2024-04-22T20:17:57,699 INFO [HiveServer2-Background-Pool: Thread-159] resource.ResourceUtils: Unable to find 'resource-types.xml'. 2024-04-22T20:17:57,704 INFO [HiveServer2-Background-Pool: Thread-159] common.TezYARNUtils: Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true 2024-04-22T20:17:57,715 INFO [HiveServer2-Background-Pool: Thread-159] Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled 2024-04-22T20:17:58,223 INFO [HiveServer2-Background-Pool: Thread-159] impl.YarnClientImpl: Submitted application application_1713840366821_0001 2024-04-22T20:17:58,226 INFO [HiveServer2-Background-Pool: Thread-159] client.TezClient: The url to track the Tez Session: http://localhost:8088/proxy/application_1713840366821_0001/{noformat} Checked application_1713840366821_0001 in logs of YARN. It's not running since cluster resource is empty, i.e. <memory:0, vCores:0> {noformat} 2024-04-22 20:17:57,916 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application 'application_1713840366821_0001' is submitted without priority hence considering default queue/cluster priority: 0 2024-04-22 20:17:57,916 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Priority '0' is acceptable in queue : default for application: application_1713840366821_0001 2024-04-22 20:17:57,935 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application with id 1 submitted by user jenkins 2024-04-22 20:17:57,935 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing application with id application_1713840366821_0001 2024-04-22 20:17:57,937 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=jenkins IP=127.0.0.1 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1713840366821_0001 QUEUENAME=default 2024-04-22 20:17:57,946 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing info for app: application_1713840366821_0001 2024-04-22 20:17:57,946 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1713840366821_0001 State change from NEW to NEW_SAVING on event = START 2024-04-22 20:17:57,947 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1713840366821_0001 State change from NEW_SAVING to SUBMITTED on event = APP_NEW_SAVED 2024-04-22 20:17:57,958 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue: Application added - appId: application_1713840366821_0001 user: jenkins leaf-queue of parent: root #applications: 1 2024-04-22 20:17:57,959 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Accepted application application_1713840366821_0001 from user: jenkins, in queue: default 2024-04-22 20:17:57,986 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1713840366821_0001 State change from SUBMITTED to ACCEPTED on event = APP_ACCEPTED 2024-04-22 20:17:58,009 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1713840366821_0001_000001 2024-04-22 20:17:58,011 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1713840366821_0001_000001 State change from NEW to SUBMITTED on event = START 2024-04-22 20:17:58,030 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractLeafQueue: Skipping activateApplications for appattempt_1713840366821_0001_000001 since cluster resource is <memory:0, vCores:0> 2024-04-22 20:17:58,030 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractLeafQueue: Application added - appId: application_1713840366821_0001 user: jenkins, leaf-queue: root.default #user-pending-applications: 1 #user-active-applications: 0 #queue-pending-applications: 1 #queue-active-applications: 0 #queue-nonrunnable-applications: 0 2024-04-22 20:17:58,030 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Added Application Attempt appattempt_1713840366821_0001_000001 to scheduler from user jenkins in queue root.default 2024-04-22 20:17:58,040 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1713840366821_0001_000001 State change from SUBMITTED to SCHEDULED on event = ATTEMPT_ADDED 2024-04-23 15:48:43,735 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: RECEIVED SIGNAL 15: SIGTERM{noformat} Then checked the logs of NodeManager. It's somehow stopped at 2024-04-22 19:46:19 and never starts again: {noformat} 2024-04-22 19:46:07,927 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered with ResourceManager as localhost:31102 with total resource of <memory:41984, vCores:8> 2024-04-22 19:46:19,394 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: RECEIVED SIGNAL 15: SIGTERM 2024-04-22 19:46:19,401 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: RECEIVED SIGNAL 15: SIGTERM 2024-04-22 19:46:19,405 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Successfully Unregistered the Node localhost:31102 with ResourceManager.{noformat} No logs of OOM Killer in dmesg found. Disk usage is low (<32%). I think we need more logs to show who stops the NodeManager. > EventsProcessorStressTest can hang > ---------------------------------- > > Key: IMPALA-13037 > URL: https://issues.apache.org/jira/browse/IMPALA-13037 > Project: IMPALA > Issue Type: Bug > Components: Catalog, Infrastructure > Reporter: Csaba Ringhofer > Priority: Critical > > The test failed with timeout. > From mvn.log the last line is: > 20:17:53 [INFO] Running > org.apache.impala.catalog.events.EventsProcessorStressTest > Things seem to be hanging from 2024.04.22 20:17:53 to 2024.04.23 > The tests seems to wait for a Hive query. > From FeSupport.INFO: > {code} > I0422 20:17:55.478875 7949 RandomHiveQueryRunner.java:1102] Client 0 running > hive query set 2: > insert into table events_stress_db_0.stress_test_tbl_0_alltypes_part > partition (year,month) select * from functional.alltypes limit 100 > create database if not exists events_stress_db_0 > drop table if exists events_stress_db_0.stress_test_tbl_0_alltypes_part > create table if not exists > events_stress_db_0.stress_test_tbl_0_alltypes_part like functional.alltypes > set hive.exec.dynamic.partition.mode = nonstrict > set hive.exec.max.dynamic.partitions = 10000 > set hive.exec.max.dynamic.partitions.pernode = 10000 > set tez.session.am.dag.submit.timeout.secs = 2 > I0422 20:17:55.478940 7949 HiveJdbcClientPool.java:102] Executing sql : > create database if not exists events_stress_db_0 > I0422 20:17:55.493497 7768 MetastoreShim.java:843] EventId: 33414 EventType: > COMMIT_TXN transaction id: 2075 > I0422 20:17:55.493682 7768 MetastoreEvents.java:302] Total number of events > received: 6 Total number of events filtered out: 0 > I0422 20:17:55.494762 7768 MetastoreEvents.java:825] EventId: 33407 > EventType: CREATE_DATABASE Successfully added database events_stress_db_0 > I0422 20:17:55.508478 7949 HiveJdbcClientPool.java:102] Executing sql : drop > table if exists events_stress_db_0.stress_test_tbl_0_alltypes_part > I0422 20:17:55.516858 7768 MetastoreEvents.java:825] EventId: 33410 > EventType: CREATE_TABLE Successfully added table > events_stress_db_0.stress_test_tbl_0_part > I0422 20:17:55.518288 7768 CatalogOpExecutor.java:4713] EventId: 33413 Table > events_stress_db_0.stress_test_tbl_0_part is not loaded. Skipping add > partitions > I0422 20:17:55.519479 7768 MetastoreEventsProcessor.java:1340] Time elapsed > in processing event batch: 178.895ms > I0422 20:17:55.521183 7768 MetastoreEventsProcessor.java:1120] Latest event > in HMS: id=33420, time=1713842275. Last synced event: id=33414, > time=1713842275. > I0422 20:17:55.533375 7949 HiveJdbcClientPool.java:102] Executing sql : > create table if not exists events_stress_db_0.stress_test_tbl_0_alltypes_part > like functional.alltypes > I0422 20:17:55.611153 7949 HiveJdbcClientPool.java:102] Executing sql : set > hive.exec.dynamic.partition.mode = nonstrict > I0422 20:17:55.616571 7949 HiveJdbcClientPool.java:102] Executing sql : set > hive.exec.max.dynamic.partitions = 10000 > I0422 20:17:55.619197 7949 HiveJdbcClientPool.java:102] Executing sql : set > hive.exec.max.dynamic.partitions.pernode = 10000 > I0422 20:17:55.621069 7949 HiveJdbcClientPool.java:102] Executing sql : set > tez.session.am.dag.submit.timeout.secs = 2 > I0422 20:17:55.622972 7949 HiveJdbcClientPool.java:102] Executing sql : > insert into table events_stress_db_0.stress_test_tbl_0_alltypes_part > partition (year,month) select * from functional.alltypes limit 100 > I0422 20:17:57.163591 7950 CatalogServiceCatalog.java:2747] Refreshing table > metadata: events_stress_db_0.stress_test_tbl_0_part > I0422 20:17:57.829802 7768 MetastoreEventsProcessor.java:982] Received 6 > events. First event id: 33416. > I0422 20:17:57.833026 7768 MetastoreShim.java:843] EventId: 33417 EventType: > COMMIT_TXN transaction id: 2076 > I0422 20:17:57.833222 7768 MetastoreShim.java:843] EventId: 33419 EventType: > COMMIT_TXN transaction id: 2077 > I0422 20:17:57.833334 7768 MetastoreShim.java:843] EventId: 33421 EventType: > COMMIT_TXN transaction id: 2078 > I0422 20:17:57.834242 7768 MetastoreShim.java:843] EventId: 33424 EventType: > COMMIT_TXN transaction id: 2079 > I0422 20:17:57.834323 7768 MetastoreEvents.java:302] Total number of events > received: 6 Total number of events filtered out: 0 > I0422 20:17:57.834570 7768 CatalogOpExecutor.java:4862] EventId: 33416 Table > events_stress_db_0.stress_test_tbl_0_part is not loaded. Not processing the > event. > I0422 20:17:57.837756 7768 MetastoreEvents.java:825] EventId: 33423 > EventType: CREATE_TABLE Successfully added table > events_stress_db_0.stress_test_tbl_0_alltypes_part > I0422 20:17:57.838668 7768 MetastoreEventsProcessor.java:1340] Time elapsed > in processing event batch: 8.625ms > I0422 20:17:57.840027 7768 MetastoreEventsProcessor.java:1120] Latest event > in HMS: id=33425, time=1713842275. Last synced event: id=33424, > time=1713842275. > I0422 20:18:03.143219 7768 MetastoreEventsProcessor.java:982] Received 0 > events. First event id: none. > E0423 15:48:25.625319 7949 RandomHiveQueryRunner.java:152] Unexpected error > received while running the hive query > Java exception follows: > java.sql.SQLException: org.apache.thrift.transport.TTransportException: > Socket is closed by peer. > at > org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:463) > at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:314) > at > org.apache.impala.testutil.HiveJdbcClientPool$HiveJdbcClient.executeSql(HiveJdbcClientPool.java:103) > at > org.apache.impala.util.RandomHiveQueryRunner$TestHiveQuery.runInternal(RandomHiveQueryRunner.java:141) > at > org.apache.impala.util.RandomHiveQueryRunner$TestHiveQuery.run(RandomHiveQueryRunner.java:150) > at > org.apache.impala.util.RandomHiveQueryRunner.lambda$start$0(RandomHiveQueryRunner.java:1104) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Suppressed: java.sql.SQLException: > org.apache.thrift.transport.TTransportException: Socket is closed by peer. > at > org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:225) > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:266) > at > org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:289) > at > org.apache.impala.testutil.HiveJdbcClientPool$HiveJdbcClient.close(HiveJdbcClientPool.java:71) > at > org.apache.impala.util.RandomHiveQueryRunner$TestHiveQuery.run(RandomHiveQueryRunner.java:151) > ... 5 more > Caused by: org.apache.thrift.transport.TTransportException: Socket is > closed by peer. > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:184) > at > org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) > at > org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:338) > at > org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:421) > at > org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:397) > at > org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39) > at > org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) > at > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464) > at > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:362) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:245) > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) > at > org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseOperation(TCLIService.java:494) > at > org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:481) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:2129) > at com.sun.proxy.$Proxy47.CloseOperation(Unknown Source) > at > org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:216) > ... 9 more > Caused by: org.apache.thrift.transport.TTransportException: Socket is closed > by peer. > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:184) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) > at > org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:338) > at > org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:421) > at > org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:397) > at > org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) > at > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464) > at > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:362) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:245) > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) > at > org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_GetOperationStatus(TCLIService.java:448) > at > org.apache.hive.service.rpc.thrift.TCLIService$Client.GetOperationStatus(TCLIService.java:435) > at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:2129) > at com.sun.proxy.$Proxy47.GetOperationStatus(Unknown Source) > at > org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:426) > ... 9 more > E0423 15:48:26.007812 7768 EventsProcessorStressTest.java:218] > java.util.concurrent.ExecutionException: Client 0 errored out while executing > query set 2 > insert into table events_stress_db_0.stress_test_tbl_0_alltypes_part > partition (year,month) select * from functional.alltypes limit 100 > create database if not exists events_stress_db_0 > drop table if exists events_stress_db_0.stress_test_tbl_0_alltypes_part > create table if not exists > events_stress_db_0.stress_test_tbl_0_alltypes_part like functional.alltypes > set hive.exec.dynamic.partition.mode = nonstrict > set hive.exec.max.dynamic.partitions = 10000 > set hive.exec.max.dynamic.partitions.pernode = 10000 > set tez.session.am.dag.submit.timeout.secs = 2 or its dependent queries. > Exception message is: org.apache.thrift.transport.TTransportException: Socket > is closed by peer. > Java exception follows: > java.util.concurrent.ExecutionException: > java.util.concurrent.ExecutionException: Client 0 errored out while executing > query set 2 > insert into table events_stress_db_0.stress_test_tbl_0_alltypes_part > partition (year,month) select * from functional.alltypes limit 100 > create database if not exists events_stress_db_0 > drop table if exists events_stress_db_0.stress_test_tbl_0_alltypes_part > create table if not exists > events_stress_db_0.stress_test_tbl_0_alltypes_part like functional.alltypes > set hive.exec.dynamic.partition.mode = nonstrict > set hive.exec.max.dynamic.partitions = 10000 > set hive.exec.max.dynamic.partitions.pernode = 10000 > set tez.session.am.dag.submit.timeout.secs = 2 or its dependent queries. > Exception message is: org.apache.thrift.transport.TTransportException: Socket > is closed by peer. > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.impala.util.RandomHiveQueryRunner.checkForErrors(RandomHiveQueryRunner.java:1130) > at > org.apache.impala.catalog.events.EventsProcessorStressTest.testUsingRandomHiveQueries(EventsProcessorStressTest.java:216) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:240) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385) > at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162) > at > org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495) > Caused by: java.util.concurrent.ExecutionException: Client 0 errored out > while executing query set 2 > insert into table events_stress_db_0.stress_test_tbl_0_alltypes_part > partition (year,month) select * from functional.alltypes limit 100 > create database if not exists events_stress_db_0 > drop table if exists events_stress_db_0.stress_test_tbl_0_alltypes_part > create table if not exists > events_stress_db_0.stress_test_tbl_0_alltypes_part like functional.alltypes > set hive.exec.dynamic.partition.mode = nonstrict > set hive.exec.max.dynamic.partitions = 10000 > set hive.exec.max.dynamic.partitions.pernode = 10000 > set tez.session.am.dag.submit.timeout.secs = 2 or its dependent queries. > Exception message is: org.apache.thrift.transport.TTransportException: Socket > is closed by peer. > at > org.apache.impala.util.RandomHiveQueryRunner.lambda$start$0(RandomHiveQueryRunner.java:1107) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.sql.SQLException: > org.apache.thrift.transport.TTransportException: Socket is closed by peer. > at > org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:463) > at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:314) > at > org.apache.impala.testutil.HiveJdbcClientPool$HiveJdbcClient.executeSql(HiveJdbcClientPool.java:103) > at > org.apache.impala.util.RandomHiveQueryRunner$TestHiveQuery.runInternal(RandomHiveQueryRunner.java:141) > at > org.apache.impala.util.RandomHiveQueryRunner$TestHiveQuery.run(RandomHiveQueryRunner.java:150) > at > org.apache.impala.util.RandomHiveQueryRunner.lambda$start$0(RandomHiveQueryRunner.java:1104) > ... 4 more > Suppressed: java.sql.SQLException: > org.apache.thrift.transport.TTransportException: Socket is closed by peer. > at > org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:225) > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:266) > at > org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:289) > at > org.apache.impala.testutil.HiveJdbcClientPool$HiveJdbcClient.close(HiveJdbcClientPool.java:71) > at > org.apache.impala.util.RandomHiveQueryRunner$TestHiveQuery.run(RandomHiveQueryRunner.java:151) > ... 5 more > Caused by: org.apache.thrift.transport.TTransportException: Socket is > closed by peer. > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:184) > at > org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) > at > org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:338) > at > org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:421) > at > org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:397) > at > org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39) > at > org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) > at > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464) > at > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:362) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:245) > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) > at > org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseOperation(TCLIService.java:494) > at > org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:481) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:2129) > at com.sun.proxy.$Proxy47.CloseOperation(Unknown Source) > at > org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:216) > ... 9 more > Caused by: org.apache.thrift.transport.TTransportException: Socket is closed > by peer. > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:184) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) > at > org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:338) > at > org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:421) > at > org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:397) > at > org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) > at > org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464) > at > org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:362) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:245) > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) > at > org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_GetOperationStatus(TCLIService.java:448) > at > org.apache.hive.service.rpc.thrift.TCLIService$Client.GetOperationStatus(TCLIService.java:435) > at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:2129) > at com.sun.proxy.$Proxy47.GetOperationStatus(Unknown Source) > at > org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:426) > ... 9 more > {code} > From hive-server2.log grepping for > jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df: > {code} > 2024-04-22T20:17:55,624 INFO [26e2776e-b6e1-4613-bc70-7a09f8adc770 > HiveServer2-Handler-Pool: Thread-121] operation.Operation: > [opType=EXECUTE_STATEMENT, > queryId=jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df, > startTime=1713842275623, sessionId=26e2776e-b6e1-4613-bc70-7a09f8adc770, > createTime=1713842273710, userName=anonymous, ipAddress=127.0.0.1] > 2024-04-22T20:17:55,625 INFO [26e2776e-b6e1-4613-bc70-7a09f8adc770 > HiveServer2-Handler-Pool: Thread-121] ql.Driver: Compiling > command(queryId=jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df): > insert into table events_stress_db_0.stress_test_tbl_0_alltypes_part > partition (year,month) select * from functional.alltypes limit 100 > 2024-04-22T20:17:55,631 INFO [26e2776e-b6e1-4613-bc70-7a09f8adc770 > HiveServer2-Handler-Pool: Thread-121] ql.QueryState: Query-level HMS cache > created for jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df > 2024-04-22T20:17:56,284 INFO [26e2776e-b6e1-4613-bc70-7a09f8adc770 > HiveServer2-Handler-Pool: Thread-121] ql.Driver: Completed compiling > command(queryId=jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df); > Time taken: 0.659 seconds > 2024-04-22T20:17:56,286 INFO [HiveServer2-Background-Pool: Thread-159] > lockmgr.DbTxnManager: Setting lock request transaction to txnid:2080 for > queryId=jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df > 2024-04-22T20:17:56,286 INFO [HiveServer2-Background-Pool: Thread-159] > lockmgr.DbLockManager: Requesting lock for > queryId=jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df > 2024-04-22T20:17:56,292 INFO [HiveServer2-Background-Pool: Thread-159] > lockmgr.DbLockManager: Response to > queryId=jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df > LockResponse(lockid:389, state:ACQUIRED) > 2024-04-22T20:17:56,293 INFO [HiveServer2-Background-Pool: Thread-159] > ql.Driver: Executing > command(queryId=jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df): > insert into table events_stress_db_0.stress_test_tbl_0_alltypes_part > partition (year,month) select * from functional.alltypes limit 100 > 2024-04-22T20:17:56,293 INFO [HiveServer2-Background-Pool: Thread-159] > ql.Driver: Query ID = > jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df > 2024-04-22T20:17:56,360 INFO [HiveServer2-Background-Pool: Thread-159] > tez.TezTask: Subscribed to counters: [] for queryId: > jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df > {code} > There are no more mentions of the query after this. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org