[
https://issues.apache.org/jira/browse/IMPALA-13037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840619#comment-17840619
]
Quanlong Huang commented on IMPALA-13037:
-----------------------------------------
Also checked logs of hive-server2. There is an application id printed by the
same thread:
{noformat}
2024-04-22T20:17:56,360 INFO [HiveServer2-Background-Pool: Thread-159]
tez.TezTask: Subscribed to counters: [] for queryId:
jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df
2024-04-22T20:17:56,360 INFO [HiveServer2-Background-Pool: Thread-159]
tez.TezTask: Tez session hasn't been created yet. Opening session
2024-04-22T20:17:56,360 INFO [HiveServer2-Background-Pool: Thread-159]
tez.TezSessionState: User of session id d6d65f07-cdff-4f5c-bbb0-b2fa24d2d1cc is
jenkins
2024-04-22T20:17:56,369 INFO [HiveServer2-Background-Pool: Thread-159]
tez.DagUtils: Localizing resource because it does not exist:
file:/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/fe/target/dependency/postgresql-42.5.1.jar
to dest:
hdfs://localhost:20500/tmp/hive/jenkins/_tez_session_dir/d6d65f07-cdff-4f5c-bbb0-b2fa24d2d1cc-resources/postgresql-42.5.1.jar
2024-04-22T20:17:56,549 INFO [HiveServer2-Background-Pool: Thread-159]
tez.DagUtils: Resource modification time: 1713842276519 for
hdfs://localhost:20500/tmp/hive/jenkins/_tez_session_dir/d6d65f07-cdff-4f5c-bbb0-b2fa24d2d1cc-resources/postgresql-42.5.1.jar
2024-04-22T20:17:56,625 INFO [HiveServer2-Background-Pool: Thread-159]
tez.TezSessionState: Created new resources: null
2024-04-22T20:17:56,627 INFO [HiveServer2-Background-Pool: Thread-159]
tez.DagUtils: Jar dir is null / directory doesn't exist. Choosing
HIVE_INSTALL_DIR - /user/jenkins/.hiveJars
2024-04-22T20:17:57,105 INFO [HiveServer2-Background-Pool: Thread-159]
tez.TezSessionState: Computed sha:
77f0dcaafc28cfe7b2d805cdf2d3a083370b2299011e98eb893bd9573e3d4c10 for file:
file:/data0/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/cdp_components-45689292/apache-hive-3.1.3000.7.2.18.0-369-bin/lib/hive-exec-3.1.3000.7.2.18.0-369.jar
of length: 74.73MB in 474 ms
2024-04-22T20:17:57,109 INFO [HiveServer2-Background-Pool: Thread-159]
tez.DagUtils: Resource modification time: 1713837749334 for
hdfs://localhost:20500/user/jenkins/.hiveJars/hive-exec-3.1.3000.7.2.18.0-369-77f0dcaafc28cfe7b2d805cdf2d3a083370b2299011e98eb893bd9573e3d4c10.jar
2024-04-22T20:17:57,227 INFO [HiveServer2-Background-Pool: Thread-159]
counters.Limits: Counter limits initialized with parameters:
GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
2024-04-22T20:17:57,227 INFO [HiveServer2-Background-Pool: Thread-159]
counters.Limits: Counter limits initialized with parameters:
GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=120
2024-04-22T20:17:57,227 INFO [HiveServer2-Background-Pool: Thread-159]
client.TezClient: Tez Client Version: [ component=tez-api,
version=0.9.1.7.2.18.0-369, revision=590a68b8a743783155fea2e6f2026f01a8775635,
SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
buildTime=2023-09-28T12:31:39Z ]
2024-04-22T20:17:57,227 INFO [HiveServer2-Background-Pool: Thread-159]
tez.TezSessionState: Opening new Tez Session (id:
d6d65f07-cdff-4f5c-bbb0-b2fa24d2d1cc, scratch dir:
hdfs://localhost:20500/tmp/hive/jenkins/_tez_session_dir/d6d65f07-cdff-4f5c-bbb0-b2fa24d2d1cc)
2024-04-22T20:17:57,293 INFO [HiveServer2-Background-Pool: Thread-159]
client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
2024-04-22T20:17:57,575 INFO [HiveServer2-Background-Pool: Thread-159]
client.TezClient: Session mode. Starting session.
2024-04-22T20:17:57,664 INFO [HiveServer2-Background-Pool: Thread-159]
client.TezClientUtils: Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is
set to true
2024-04-22T20:17:57,675 INFO [HiveServer2-Background-Pool: Thread-159]
client.TezClient: Tez system stage directory
hdfs://localhost:20500/tmp/hive/jenkins/_tez_session_dir/d6d65f07-cdff-4f5c-bbb0-b2fa24d2d1cc/.tez/application_1713840366821_0001
doesn't exist and is created
2024-04-22T20:17:57,699 INFO [HiveServer2-Background-Pool: Thread-159]
conf.Configuration: resource-types.xml not found
2024-04-22T20:17:57,699 INFO [HiveServer2-Background-Pool: Thread-159]
resource.ResourceUtils: Unable to find 'resource-types.xml'.
2024-04-22T20:17:57,704 INFO [HiveServer2-Background-Pool: Thread-159]
common.TezYARNUtils: Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is
set to true
2024-04-22T20:17:57,715 INFO [HiveServer2-Background-Pool: Thread-159]
Configuration.deprecation:
yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead,
use yarn.system-metrics-publisher.enabled
2024-04-22T20:17:58,223 INFO [HiveServer2-Background-Pool: Thread-159]
impl.YarnClientImpl: Submitted application application_1713840366821_0001
2024-04-22T20:17:58,226 INFO [HiveServer2-Background-Pool: Thread-159]
client.TezClient: The url to track the Tez Session:
http://localhost:8088/proxy/application_1713840366821_0001/{noformat}
Checked application_1713840366821_0001 in logs of YARN. It's not running since
cluster resource is empty, i.e. <memory:0, vCores:0>
{noformat}
2024-04-22 20:17:57,916 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Application 'application_1713840366821_0001' is submitted without priority
hence considering default queue/cluster priority: 0
2024-04-22 20:17:57,916 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Priority '0' is acceptable in queue : default for application:
application_1713840366821_0001
2024-04-22 20:17:57,935 INFO
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application with
id 1 submitted by user jenkins
2024-04-22 20:17:57,935 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing
application with id application_1713840366821_0001
2024-04-22 20:17:57,937 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=jenkins
IP=127.0.0.1 OPERATION=Submit Application Request TARGET=ClientRMService
RESULT=SUCCESS APPID=application_1713840366821_0001 QUEUENAME=default
2024-04-22 20:17:57,946 INFO
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing
info for app: application_1713840366821_0001
2024-04-22 20:17:57,946 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1713840366821_0001 State change from NEW to NEW_SAVING on event =
START
2024-04-22 20:17:57,947 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1713840366821_0001 State change from NEW_SAVING to SUBMITTED on
event = APP_NEW_SAVED
2024-04-22 20:17:57,958 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue:
Application added - appId: application_1713840366821_0001 user: jenkins
leaf-queue of parent: root #applications: 1
2024-04-22 20:17:57,959 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Accepted application application_1713840366821_0001 from user: jenkins, in
queue: default
2024-04-22 20:17:57,986 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1713840366821_0001 State change from SUBMITTED to ACCEPTED on event
= APP_ACCEPTED
2024-04-22 20:17:58,009 INFO
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
Registering app attempt : appattempt_1713840366821_0001_000001
2024-04-22 20:17:58,011 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1713840366821_0001_000001 State change from NEW to SUBMITTED on
event = START
2024-04-22 20:17:58,030 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractLeafQueue:
Skipping activateApplications for appattempt_1713840366821_0001_000001 since
cluster resource is <memory:0, vCores:0>
2024-04-22 20:17:58,030 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractLeafQueue:
Application added - appId: application_1713840366821_0001 user: jenkins,
leaf-queue: root.default #user-pending-applications: 1
#user-active-applications: 0 #queue-pending-applications: 1
#queue-active-applications: 0 #queue-nonrunnable-applications: 0
2024-04-22 20:17:58,030 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Added Application Attempt appattempt_1713840366821_0001_000001 to scheduler
from user jenkins in queue root.default
2024-04-22 20:17:58,040 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1713840366821_0001_000001 State change from SUBMITTED to SCHEDULED
on event = ATTEMPT_ADDED
2024-04-23 15:48:43,735 ERROR
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: RECEIVED SIGNAL
15: SIGTERM{noformat}
Then checked the logs of NodeManager. It's somehow stopped at 2024-04-22
19:46:19 and never starts again:
{noformat}
2024-04-22 19:46:07,927 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered
with ResourceManager as localhost:31102 with total resource of <memory:41984,
vCores:8>
2024-04-22 19:46:19,394 ERROR
org.apache.hadoop.yarn.server.nodemanager.NodeManager: RECEIVED SIGNAL 15:
SIGTERM
2024-04-22 19:46:19,401 ERROR
org.apache.hadoop.yarn.server.nodemanager.NodeManager: RECEIVED SIGNAL 15:
SIGTERM
2024-04-22 19:46:19,405 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Successfully
Unregistered the Node localhost:31102 with ResourceManager.{noformat}
No logs of OOM Killer in dmesg found. Disk usage is low (<32%). I think we need
more logs to show who stops the NodeManager.
> EventsProcessorStressTest can hang
> ----------------------------------
>
> Key: IMPALA-13037
> URL: https://issues.apache.org/jira/browse/IMPALA-13037
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Infrastructure
> Reporter: Csaba Ringhofer
> Priority: Critical
>
> The test failed with timeout.
> From mvn.log the last line is:
> 20:17:53 [INFO] Running
> org.apache.impala.catalog.events.EventsProcessorStressTest
> Things seem to be hanging from 2024.04.22 20:17:53 to 2024.04.23
> The tests seems to wait for a Hive query.
> From FeSupport.INFO:
> {code}
> I0422 20:17:55.478875 7949 RandomHiveQueryRunner.java:1102] Client 0 running
> hive query set 2:
> insert into table events_stress_db_0.stress_test_tbl_0_alltypes_part
> partition (year,month) select * from functional.alltypes limit 100
> create database if not exists events_stress_db_0
> drop table if exists events_stress_db_0.stress_test_tbl_0_alltypes_part
> create table if not exists
> events_stress_db_0.stress_test_tbl_0_alltypes_part like functional.alltypes
> set hive.exec.dynamic.partition.mode = nonstrict
> set hive.exec.max.dynamic.partitions = 10000
> set hive.exec.max.dynamic.partitions.pernode = 10000
> set tez.session.am.dag.submit.timeout.secs = 2
> I0422 20:17:55.478940 7949 HiveJdbcClientPool.java:102] Executing sql :
> create database if not exists events_stress_db_0
> I0422 20:17:55.493497 7768 MetastoreShim.java:843] EventId: 33414 EventType:
> COMMIT_TXN transaction id: 2075
> I0422 20:17:55.493682 7768 MetastoreEvents.java:302] Total number of events
> received: 6 Total number of events filtered out: 0
> I0422 20:17:55.494762 7768 MetastoreEvents.java:825] EventId: 33407
> EventType: CREATE_DATABASE Successfully added database events_stress_db_0
> I0422 20:17:55.508478 7949 HiveJdbcClientPool.java:102] Executing sql : drop
> table if exists events_stress_db_0.stress_test_tbl_0_alltypes_part
> I0422 20:17:55.516858 7768 MetastoreEvents.java:825] EventId: 33410
> EventType: CREATE_TABLE Successfully added table
> events_stress_db_0.stress_test_tbl_0_part
> I0422 20:17:55.518288 7768 CatalogOpExecutor.java:4713] EventId: 33413 Table
> events_stress_db_0.stress_test_tbl_0_part is not loaded. Skipping add
> partitions
> I0422 20:17:55.519479 7768 MetastoreEventsProcessor.java:1340] Time elapsed
> in processing event batch: 178.895ms
> I0422 20:17:55.521183 7768 MetastoreEventsProcessor.java:1120] Latest event
> in HMS: id=33420, time=1713842275. Last synced event: id=33414,
> time=1713842275.
> I0422 20:17:55.533375 7949 HiveJdbcClientPool.java:102] Executing sql :
> create table if not exists events_stress_db_0.stress_test_tbl_0_alltypes_part
> like functional.alltypes
> I0422 20:17:55.611153 7949 HiveJdbcClientPool.java:102] Executing sql : set
> hive.exec.dynamic.partition.mode = nonstrict
> I0422 20:17:55.616571 7949 HiveJdbcClientPool.java:102] Executing sql : set
> hive.exec.max.dynamic.partitions = 10000
> I0422 20:17:55.619197 7949 HiveJdbcClientPool.java:102] Executing sql : set
> hive.exec.max.dynamic.partitions.pernode = 10000
> I0422 20:17:55.621069 7949 HiveJdbcClientPool.java:102] Executing sql : set
> tez.session.am.dag.submit.timeout.secs = 2
> I0422 20:17:55.622972 7949 HiveJdbcClientPool.java:102] Executing sql :
> insert into table events_stress_db_0.stress_test_tbl_0_alltypes_part
> partition (year,month) select * from functional.alltypes limit 100
> I0422 20:17:57.163591 7950 CatalogServiceCatalog.java:2747] Refreshing table
> metadata: events_stress_db_0.stress_test_tbl_0_part
> I0422 20:17:57.829802 7768 MetastoreEventsProcessor.java:982] Received 6
> events. First event id: 33416.
> I0422 20:17:57.833026 7768 MetastoreShim.java:843] EventId: 33417 EventType:
> COMMIT_TXN transaction id: 2076
> I0422 20:17:57.833222 7768 MetastoreShim.java:843] EventId: 33419 EventType:
> COMMIT_TXN transaction id: 2077
> I0422 20:17:57.833334 7768 MetastoreShim.java:843] EventId: 33421 EventType:
> COMMIT_TXN transaction id: 2078
> I0422 20:17:57.834242 7768 MetastoreShim.java:843] EventId: 33424 EventType:
> COMMIT_TXN transaction id: 2079
> I0422 20:17:57.834323 7768 MetastoreEvents.java:302] Total number of events
> received: 6 Total number of events filtered out: 0
> I0422 20:17:57.834570 7768 CatalogOpExecutor.java:4862] EventId: 33416 Table
> events_stress_db_0.stress_test_tbl_0_part is not loaded. Not processing the
> event.
> I0422 20:17:57.837756 7768 MetastoreEvents.java:825] EventId: 33423
> EventType: CREATE_TABLE Successfully added table
> events_stress_db_0.stress_test_tbl_0_alltypes_part
> I0422 20:17:57.838668 7768 MetastoreEventsProcessor.java:1340] Time elapsed
> in processing event batch: 8.625ms
> I0422 20:17:57.840027 7768 MetastoreEventsProcessor.java:1120] Latest event
> in HMS: id=33425, time=1713842275. Last synced event: id=33424,
> time=1713842275.
> I0422 20:18:03.143219 7768 MetastoreEventsProcessor.java:982] Received 0
> events. First event id: none.
> E0423 15:48:25.625319 7949 RandomHiveQueryRunner.java:152] Unexpected error
> received while running the hive query
> Java exception follows:
> java.sql.SQLException: org.apache.thrift.transport.TTransportException:
> Socket is closed by peer.
> at
> org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:463)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:314)
> at
> org.apache.impala.testutil.HiveJdbcClientPool$HiveJdbcClient.executeSql(HiveJdbcClientPool.java:103)
> at
> org.apache.impala.util.RandomHiveQueryRunner$TestHiveQuery.runInternal(RandomHiveQueryRunner.java:141)
> at
> org.apache.impala.util.RandomHiveQueryRunner$TestHiveQuery.run(RandomHiveQueryRunner.java:150)
> at
> org.apache.impala.util.RandomHiveQueryRunner.lambda$start$0(RandomHiveQueryRunner.java:1104)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Suppressed: java.sql.SQLException:
> org.apache.thrift.transport.TTransportException: Socket is closed by peer.
> at
> org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:225)
> at
> org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:266)
> at
> org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:289)
> at
> org.apache.impala.testutil.HiveJdbcClientPool$HiveJdbcClient.close(HiveJdbcClientPool.java:71)
> at
> org.apache.impala.util.RandomHiveQueryRunner$TestHiveQuery.run(RandomHiveQueryRunner.java:151)
> ... 5 more
> Caused by: org.apache.thrift.transport.TTransportException: Socket is
> closed by peer.
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:184)
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:338)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:421)
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:397)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:362)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:245)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseOperation(TCLIService.java:494)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:481)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:2129)
> at com.sun.proxy.$Proxy47.CloseOperation(Unknown Source)
> at
> org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:216)
> ... 9 more
> Caused by: org.apache.thrift.transport.TTransportException: Socket is closed
> by peer.
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:184)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:338)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:421)
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:397)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:362)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:245)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_GetOperationStatus(TCLIService.java:448)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.GetOperationStatus(TCLIService.java:435)
> at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:2129)
> at com.sun.proxy.$Proxy47.GetOperationStatus(Unknown Source)
> at
> org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:426)
> ... 9 more
> E0423 15:48:26.007812 7768 EventsProcessorStressTest.java:218]
> java.util.concurrent.ExecutionException: Client 0 errored out while executing
> query set 2
> insert into table events_stress_db_0.stress_test_tbl_0_alltypes_part
> partition (year,month) select * from functional.alltypes limit 100
> create database if not exists events_stress_db_0
> drop table if exists events_stress_db_0.stress_test_tbl_0_alltypes_part
> create table if not exists
> events_stress_db_0.stress_test_tbl_0_alltypes_part like functional.alltypes
> set hive.exec.dynamic.partition.mode = nonstrict
> set hive.exec.max.dynamic.partitions = 10000
> set hive.exec.max.dynamic.partitions.pernode = 10000
> set tez.session.am.dag.submit.timeout.secs = 2 or its dependent queries.
> Exception message is: org.apache.thrift.transport.TTransportException: Socket
> is closed by peer.
> Java exception follows:
> java.util.concurrent.ExecutionException:
> java.util.concurrent.ExecutionException: Client 0 errored out while executing
> query set 2
> insert into table events_stress_db_0.stress_test_tbl_0_alltypes_part
> partition (year,month) select * from functional.alltypes limit 100
> create database if not exists events_stress_db_0
> drop table if exists events_stress_db_0.stress_test_tbl_0_alltypes_part
> create table if not exists
> events_stress_db_0.stress_test_tbl_0_alltypes_part like functional.alltypes
> set hive.exec.dynamic.partition.mode = nonstrict
> set hive.exec.max.dynamic.partitions = 10000
> set hive.exec.max.dynamic.partitions.pernode = 10000
> set tez.session.am.dag.submit.timeout.secs = 2 or its dependent queries.
> Exception message is: org.apache.thrift.transport.TTransportException: Socket
> is closed by peer.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.impala.util.RandomHiveQueryRunner.checkForErrors(RandomHiveQueryRunner.java:1130)
> at
> org.apache.impala.catalog.events.EventsProcessorStressTest.testUsingRandomHiveQueries(EventsProcessorStressTest.java:216)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:240)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
> at
> org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
> Caused by: java.util.concurrent.ExecutionException: Client 0 errored out
> while executing query set 2
> insert into table events_stress_db_0.stress_test_tbl_0_alltypes_part
> partition (year,month) select * from functional.alltypes limit 100
> create database if not exists events_stress_db_0
> drop table if exists events_stress_db_0.stress_test_tbl_0_alltypes_part
> create table if not exists
> events_stress_db_0.stress_test_tbl_0_alltypes_part like functional.alltypes
> set hive.exec.dynamic.partition.mode = nonstrict
> set hive.exec.max.dynamic.partitions = 10000
> set hive.exec.max.dynamic.partitions.pernode = 10000
> set tez.session.am.dag.submit.timeout.secs = 2 or its dependent queries.
> Exception message is: org.apache.thrift.transport.TTransportException: Socket
> is closed by peer.
> at
> org.apache.impala.util.RandomHiveQueryRunner.lambda$start$0(RandomHiveQueryRunner.java:1107)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.sql.SQLException:
> org.apache.thrift.transport.TTransportException: Socket is closed by peer.
> at
> org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:463)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:314)
> at
> org.apache.impala.testutil.HiveJdbcClientPool$HiveJdbcClient.executeSql(HiveJdbcClientPool.java:103)
> at
> org.apache.impala.util.RandomHiveQueryRunner$TestHiveQuery.runInternal(RandomHiveQueryRunner.java:141)
> at
> org.apache.impala.util.RandomHiveQueryRunner$TestHiveQuery.run(RandomHiveQueryRunner.java:150)
> at
> org.apache.impala.util.RandomHiveQueryRunner.lambda$start$0(RandomHiveQueryRunner.java:1104)
> ... 4 more
> Suppressed: java.sql.SQLException:
> org.apache.thrift.transport.TTransportException: Socket is closed by peer.
> at
> org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:225)
> at
> org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:266)
> at
> org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:289)
> at
> org.apache.impala.testutil.HiveJdbcClientPool$HiveJdbcClient.close(HiveJdbcClientPool.java:71)
> at
> org.apache.impala.util.RandomHiveQueryRunner$TestHiveQuery.run(RandomHiveQueryRunner.java:151)
> ... 5 more
> Caused by: org.apache.thrift.transport.TTransportException: Socket is
> closed by peer.
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:184)
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:338)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:421)
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:397)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:362)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:245)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseOperation(TCLIService.java:494)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:481)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:2129)
> at com.sun.proxy.$Proxy47.CloseOperation(Unknown Source)
> at
> org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:216)
> ... 9 more
> Caused by: org.apache.thrift.transport.TTransportException: Socket is closed
> by peer.
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:184)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:338)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:421)
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:397)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:362)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:245)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_GetOperationStatus(TCLIService.java:448)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.GetOperationStatus(TCLIService.java:435)
> at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:2129)
> at com.sun.proxy.$Proxy47.GetOperationStatus(Unknown Source)
> at
> org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:426)
> ... 9 more
> {code}
> From hive-server2.log grepping for
> jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df:
> {code}
> 2024-04-22T20:17:55,624 INFO [26e2776e-b6e1-4613-bc70-7a09f8adc770
> HiveServer2-Handler-Pool: Thread-121] operation.Operation:
> [opType=EXECUTE_STATEMENT,
> queryId=jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df,
> startTime=1713842275623, sessionId=26e2776e-b6e1-4613-bc70-7a09f8adc770,
> createTime=1713842273710, userName=anonymous, ipAddress=127.0.0.1]
> 2024-04-22T20:17:55,625 INFO [26e2776e-b6e1-4613-bc70-7a09f8adc770
> HiveServer2-Handler-Pool: Thread-121] ql.Driver: Compiling
> command(queryId=jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df):
> insert into table events_stress_db_0.stress_test_tbl_0_alltypes_part
> partition (year,month) select * from functional.alltypes limit 100
> 2024-04-22T20:17:55,631 INFO [26e2776e-b6e1-4613-bc70-7a09f8adc770
> HiveServer2-Handler-Pool: Thread-121] ql.QueryState: Query-level HMS cache
> created for jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df
> 2024-04-22T20:17:56,284 INFO [26e2776e-b6e1-4613-bc70-7a09f8adc770
> HiveServer2-Handler-Pool: Thread-121] ql.Driver: Completed compiling
> command(queryId=jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df);
> Time taken: 0.659 seconds
> 2024-04-22T20:17:56,286 INFO [HiveServer2-Background-Pool: Thread-159]
> lockmgr.DbTxnManager: Setting lock request transaction to txnid:2080 for
> queryId=jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df
> 2024-04-22T20:17:56,286 INFO [HiveServer2-Background-Pool: Thread-159]
> lockmgr.DbLockManager: Requesting lock for
> queryId=jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df
> 2024-04-22T20:17:56,292 INFO [HiveServer2-Background-Pool: Thread-159]
> lockmgr.DbLockManager: Response to
> queryId=jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df
> LockResponse(lockid:389, state:ACQUIRED)
> 2024-04-22T20:17:56,293 INFO [HiveServer2-Background-Pool: Thread-159]
> ql.Driver: Executing
> command(queryId=jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df):
> insert into table events_stress_db_0.stress_test_tbl_0_alltypes_part
> partition (year,month) select * from functional.alltypes limit 100
> 2024-04-22T20:17:56,293 INFO [HiveServer2-Background-Pool: Thread-159]
> ql.Driver: Query ID =
> jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df
> 2024-04-22T20:17:56,360 INFO [HiveServer2-Background-Pool: Thread-159]
> tez.TezTask: Subscribed to counters: [] for queryId:
> jenkins_20240422201755_96876acb-ee10-409e-a6da-bd1a9b4bc6df
> {code}
> There are no more mentions of the query after this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
