[
https://issues.apache.org/jira/browse/HIVE-20209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20209:
------------------------------------
Description:
Run the following command:
{code:java}
repl dump `*` from 60758 with ('hive.repl.dump.metadata.only'='true',
'hive.repl.dump.include.acid.tables'='true');
{code}
See this in hs2.log:
{code:java}
2018-07-10T18:07:32,308 INFO [HiveServer2-Handler-Pool: Thread-14380]:
conf.HiveConf (HiveConf.java:getLogIdVar(5061)) - Using the default value
passed in for log id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:32,309 INFO [HiveServer2-Handler-Pool: Thread-14380]:
session.SessionState (:()) - Updating thread name to
f1e13736-3f10-4abf-a29b-683b534dfa4c HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:32,311 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: operation.OperationManager (:()) -
Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=16eb1d07-e125-490c-8ab8-90192bfd459b]
2018-07-10T18:07:32,314 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: ql.Driver (:()) - Compiling
command(queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b): repl
dump `*` from 60758 with ('hive.repl.dump.metadata.only'='true',
'hive.repl.dump.include.acid.tables'='true')
2018-07-10T18:07:32,317 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: metastore.HiveMetaStoreClient (:()) -
Trying to connect to metastore with URI
thrift://hwx-demo-2.field.hortonworks.com:9083
2018-07-10T18:07:32,317 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: metastore.HiveMetaStoreClient (:()) -
Opened a connection to metastore, current connections: 19
2018-07-10T18:07:32,319 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: metastore.HiveMetaStoreClient (:()) -
Connected to metastore.
2018-07-10T18:07:32,319 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: metastore.RetryingMetaStoreClient
(:()) - RetryingMetaStoreClient proxy=class
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive
(auth:SIMPLE) retries=24 delay=5 lifetime=0
2018-07-10T18:07:32,439 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: ql.Driver (:()) - Semantic Analysis
Completed (retrial = false)
2018-07-10T18:07:32,440 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: ql.Driver (:()) - Returning Hive
schema: Schema(fieldSchemas:[FieldSchema(name:dump_dir, type:string,
comment:from deserializer), FieldSchema(name:last_repl_id, type:string,
comment:from deserializer)], properties:null)
2018-07-10T18:07:32,443 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: exec.ListSinkOperator (:()) -
Initializing operator LIST_SINK[0]
2018-07-10T18:07:32,446 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: ql.Driver (:()) - Completed compiling
command(queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b); Time
taken: 0.132 seconds
2018-07-10T18:07:32,447 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: conf.HiveConf
(HiveConf.java:getLogIdVar(5061)) - Using the default value passed in for log
id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:32,448 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: session.SessionState (:()) - Resetting
thread name to HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:32,451 INFO [HiveServer2-Background-Pool: Thread-15161]:
reexec.ReExecDriver (:()) - Execution #1 of query
2018-07-10T18:07:32,452 INFO [HiveServer2-Background-Pool: Thread-15161]:
lockmgr.DbTxnManager (:()) - Setting lock request transaction to txnid:30327
for queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b
2018-07-10T18:07:32,454 INFO [HiveServer2-Background-Pool: Thread-15161]:
lockmgr.DbLockManager (:()) - Requesting:
queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b
LockRequest(component:[LockComponent(type:SHARED_READ, level:DB,
dbname:default, operationType:SELECT), LockComponent(type:SHARED_READ,
level:DB, dbname:hwxdemo, operationType:SELECT),
LockComponent(type:SHARED_READ, level:DB, dbname:information_schema,
operationType:SELECT), LockComponent(type:SHARED_READ, level:DB, dbname:sys,
operationType:SELECT)], txnid:30327, user:hive,
hostname:hwx-demo-2.field.hortonworks.com,
agentInfo:hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b)
2018-07-10T18:07:32,497 INFO [HiveServer2-Background-Pool: Thread-15161]:
lockmgr.DbLockManager (:()) - Response to
queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b
LockResponse(lockid:30305, state:ACQUIRED)
2018-07-10T18:07:32,501 INFO [HiveServer2-Background-Pool: Thread-15161]:
ql.Driver (:()) - Executing
command(queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b): repl
dump `*` from 60758 with ('hive.repl.dump.metadata.only'='true',
'hive.repl.dump.include.acid.tables'='true')
2018-07-10T18:07:32,503 INFO [HiveServer2-Background-Pool: Thread-15161]:
ql.Driver (:()) - Starting task [Stage-0:REPL_DUMP] in serial mode
2018-07-10T18:07:32,506 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore, current
connections: 18
2018-07-10T18:07:32,509 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
thrift://hwx-demo-2.field.hortonworks.com:9083
2018-07-10T18:07:32,509 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore, current
connections: 19
2018-07-10T18:07:32,510 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Connected to metastore.
2018-07-10T18:07:32,510 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient proxy=class
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive
(auth:SIMPLE) retries=24 delay=5 lifetime=0
2018-07-10T18:07:32,536 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore, current
connections: 18
2018-07-10T18:07:32,538 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
thrift://hwx-demo-2.field.hortonworks.com:9083
2018-07-10T18:07:32,539 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore, current
connections: 19
2018-07-10T18:07:32,542 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Connected to metastore.
2018-07-10T18:07:32,542 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient proxy=class
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive
(auth:SIMPLE) retries=24 delay=5 lifetime=0
2018-07-10T18:07:32,565 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::START:
\{"dbName":"*","dumpType":"INCREMENTAL","estimatedNumEvents":0,"dumpStartTime":1531246052}
2018-07-10T18:07:32,577 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore, current
connections: 18
2018-07-10T18:07:32,578 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60759 COMMIT_TXN message :
\{"txnid":30321,"timestamp":1531246017,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,603 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60759","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"1/0","dumpTime":1531246052}
2018-07-10T18:07:32,604 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60760 COMMIT_TXN message :
\{"txnid":30322,"timestamp":1531246021,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,628 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60760","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"2/0","dumpTime":1531246052}
2018-07-10T18:07:32,629 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60761 OPEN_TXN message :
\{"txnIds":null,"timestamp":1531246022,"fromTxnId":30323,"toTxnId":30323,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,653 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60761","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"3/0","dumpTime":1531246052}
2018-07-10T18:07:32,654 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60762 COMMIT_TXN message :
\{"txnid":30323,"timestamp":1531246028,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,679 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60762","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"4/0","dumpTime":1531246052}
2018-07-10T18:07:32,680 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60763 OPEN_TXN message :
\{"txnIds":null,"timestamp":1531246033,"fromTxnId":30324,"toTxnId":30324,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,702 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60763","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"5/0","dumpTime":1531246052}
2018-07-10T18:07:32,702 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60764 OPEN_TXN message :
\{"txnIds":null,"timestamp":1531246038,"fromTxnId":30325,"toTxnId":30325,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,724 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60764","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"6/0","dumpTime":1531246052}
2018-07-10T18:07:32,725 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60765 COMMIT_TXN message :
\{"txnid":30324,"timestamp":1531246038,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,747 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60765","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"7/0","dumpTime":1531246052}
2018-07-10T18:07:32,748 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60766 COMMIT_TXN message :
\{"txnid":30325,"timestamp":1531246043,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,768 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60766","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"8/0","dumpTime":1531246052}
2018-07-10T18:07:32,769 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60767 OPEN_TXN message :
\{"txnIds":null,"timestamp":1531246043,"fromTxnId":30326,"toTxnId":30326,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,791 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60767","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"9/0","dumpTime":1531246052}
2018-07-10T18:07:32,791 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60768 COMMIT_TXN message :
\{"txnid":30326,"timestamp":1531246049,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,816 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60768","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"10/0","dumpTime":1531246052}
2018-07-10T18:07:32,817 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60769 OPEN_TXN message :
\{"txnIds":null,"timestamp":1531246052,"fromTxnId":30327,"toTxnId":30327,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,841 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60769","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"11/0","dumpTime":1531246052}
2018-07-10T18:07:32,841 WARN [HiveServer2-Background-Pool: Thread-15161]:
metastore.RetryingMetaStoreClient (:()) - MetaStoreClient lost connection.
Attempting to reconnect (1 of 24) after 5s. getNextNotification
org.apache.thrift.transport.TTransportException: Cannot write to null
outputStream
at
org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:178)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:106)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:70)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_next_notification(ThriftHiveMetastore.java:5547)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_next_notification(ThriftHiveMetastore.java:5539)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getNextNotification(HiveMetaStoreClient.java:2697)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at com.sun.proxy.$Proxy60.getNextNotification(Unknown Source) ~[?:?]
at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2773)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at com.sun.proxy.$Proxy60.getNextNotification(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.metastore.messaging.EventUtils$MSClientNotificationFetcher.getNextNotificationEvents(EventUtils.java:94)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hadoop.hive.metastore.messaging.EventUtils$NotificationEventIterator.fetchNextBatch(EventUtils.java:146)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hadoop.hive.metastore.messaging.EventUtils$NotificationEventIterator.hasNext(EventUtils.java:176)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.incrementalDump(ReplDumpTask.java:170)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.execute(ReplDumpTask.java:121)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2668)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2339)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2015)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1713)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1707)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)
~[hive-service-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
~[hive-service-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)
~[hive-service-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
~[hadoop-common-3.1.0.3.0.0.0-1605.jar:?]
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:329)
~[hive-service-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_112]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_112]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[?:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
2018-07-10T18:07:33,157 INFO [org.apache.ranger.audit.queue.AuditBatchQueue1]:
destination.HDFSAuditDestination (:()) - Flushing HDFS audit. Event Size:4
2018-07-10T18:07:37,842 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient trying
reconnect as hive (auth:SIMPLE)
2018-07-10T18:07:37,842 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
thrift://hwx-demo-2.field.hortonworks.com:9083
2018-07-10T18:07:37,842 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore, current
connections: 19
2018-07-10T18:07:37,843 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Connected to metastore.
2018-07-10T18:07:37,864 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::END:
\{"dbName":"*","dumpType":"INCREMENTAL","actualNumEvents":11,"dumpEndTime":1531246057,"dumpDir":"/user/hive/repl/c72dc152-c252-4454-9b0e-cf8c335ab07c","lastReplId":"60769"}
2018-07-10T18:07:37,864 INFO [HiveServer2-Background-Pool: Thread-15161]:
repl.ReplDumpTask (:()) - Done dumping events, preparing to return
/user/hive/repl/c72dc152-c252-4454-9b0e-cf8c335ab07c,60769
2018-07-10T18:07:37,919 INFO [HiveServer2-Background-Pool: Thread-15161]:
ql.Driver (:()) - Completed executing
command(queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b); Time
taken: 5.419 seconds
2018-07-10T18:07:37,919 INFO [HiveServer2-Background-Pool: Thread-15161]:
ql.Driver (:()) - OK
{code}
Metastore connection failed 1st attempt, but success after reconnect. That adds
5s for every repl dump command and likely to leak connection.
Similarly, Hive.close() also causes
{code:java}
org.apache.thrift.transport.TTransportException: Cannot write to null
outputStream{code}
was:
Run the following command:
{code}
repl dump `*` from 60758 with ('hive.repl.dump.metadata.only'='true',
'hive.repl.dump.include.acid.tables'='true');
{code}
See this in hs2.log:
{code}
2018-07-10T18:07:32,308 INFO [HiveServer2-Handler-Pool: Thread-14380]:
conf.HiveConf (HiveConf.java:getLogIdVar(5061)) - Using the default value
passed in for log id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:32,309 INFO [HiveServer2-Handler-Pool: Thread-14380]:
session.SessionState (:()) - Updating thread name to
f1e13736-3f10-4abf-a29b-683b534dfa4c HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:32,311 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: operation.OperationManager (:()) -
Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=16eb1d07-e125-490c-8ab8-90192bfd459b]
2018-07-10T18:07:32,314 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: ql.Driver (:()) - Compiling
command(queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b): repl
dump `*` from 60758 with ('hive.repl.dump.metadata.only'='true',
'hive.repl.dump.include.acid.tables'='true')
2018-07-10T18:07:32,317 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: metastore.HiveMetaStoreClient (:()) -
Trying to connect to metastore with URI
thrift://hwx-demo-2.field.hortonworks.com:9083
2018-07-10T18:07:32,317 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: metastore.HiveMetaStoreClient (:()) -
Opened a connection to metastore, current connections: 19
2018-07-10T18:07:32,319 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: metastore.HiveMetaStoreClient (:()) -
Connected to metastore.
2018-07-10T18:07:32,319 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: metastore.RetryingMetaStoreClient
(:()) - RetryingMetaStoreClient proxy=class
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive
(auth:SIMPLE) retries=24 delay=5 lifetime=0
2018-07-10T18:07:32,439 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: ql.Driver (:()) - Semantic Analysis
Completed (retrial = false)
2018-07-10T18:07:32,440 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: ql.Driver (:()) - Returning Hive
schema: Schema(fieldSchemas:[FieldSchema(name:dump_dir, type:string,
comment:from deserializer), FieldSchema(name:last_repl_id, type:string,
comment:from deserializer)], properties:null)
2018-07-10T18:07:32,443 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: exec.ListSinkOperator (:()) -
Initializing operator LIST_SINK[0]
2018-07-10T18:07:32,446 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: ql.Driver (:()) - Completed compiling
command(queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b); Time
taken: 0.132 seconds
2018-07-10T18:07:32,447 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: conf.HiveConf
(HiveConf.java:getLogIdVar(5061)) - Using the default value passed in for log
id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:32,448 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: session.SessionState (:()) - Resetting
thread name to HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:32,451 INFO [HiveServer2-Background-Pool: Thread-15161]:
reexec.ReExecDriver (:()) - Execution #1 of query
2018-07-10T18:07:32,452 INFO [HiveServer2-Background-Pool: Thread-15161]:
lockmgr.DbTxnManager (:()) - Setting lock request transaction to txnid:30327
for queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b
2018-07-10T18:07:32,454 INFO [HiveServer2-Background-Pool: Thread-15161]:
lockmgr.DbLockManager (:()) - Requesting:
queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b
LockRequest(component:[LockComponent(type:SHARED_READ, level:DB,
dbname:default, operationType:SELECT), LockComponent(type:SHARED_READ,
level:DB, dbname:hwxdemo, operationType:SELECT),
LockComponent(type:SHARED_READ, level:DB, dbname:information_schema,
operationType:SELECT), LockComponent(type:SHARED_READ, level:DB, dbname:sys,
operationType:SELECT)], txnid:30327, user:hive,
hostname:hwx-demo-2.field.hortonworks.com,
agentInfo:hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b)
2018-07-10T18:07:32,497 INFO [HiveServer2-Background-Pool: Thread-15161]:
lockmgr.DbLockManager (:()) - Response to
queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b
LockResponse(lockid:30305, state:ACQUIRED)
2018-07-10T18:07:32,501 INFO [HiveServer2-Background-Pool: Thread-15161]:
ql.Driver (:()) - Executing
command(queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b): repl
dump `*` from 60758 with ('hive.repl.dump.metadata.only'='true',
'hive.repl.dump.include.acid.tables'='true')
2018-07-10T18:07:32,503 INFO [HiveServer2-Background-Pool: Thread-15161]:
ql.Driver (:()) - Starting task [Stage-0:REPL_DUMP] in serial mode
2018-07-10T18:07:32,506 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore, current
connections: 18
2018-07-10T18:07:32,509 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
thrift://hwx-demo-2.field.hortonworks.com:9083
2018-07-10T18:07:32,509 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore, current
connections: 19
2018-07-10T18:07:32,510 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Connected to metastore.
2018-07-10T18:07:32,510 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient proxy=class
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive
(auth:SIMPLE) retries=24 delay=5 lifetime=0
2018-07-10T18:07:32,536 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore, current
connections: 18
2018-07-10T18:07:32,538 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
thrift://hwx-demo-2.field.hortonworks.com:9083
2018-07-10T18:07:32,539 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore, current
connections: 19
2018-07-10T18:07:32,542 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Connected to metastore.
2018-07-10T18:07:32,542 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient proxy=class
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive
(auth:SIMPLE) retries=24 delay=5 lifetime=0
2018-07-10T18:07:32,565 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::START:
\{"dbName":"*","dumpType":"INCREMENTAL","estimatedNumEvents":0,"dumpStartTime":1531246052}
2018-07-10T18:07:32,577 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore, current
connections: 18
2018-07-10T18:07:32,578 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60759 COMMIT_TXN message :
\{"txnid":30321,"timestamp":1531246017,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,603 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60759","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"1/0","dumpTime":1531246052}
2018-07-10T18:07:32,604 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60760 COMMIT_TXN message :
\{"txnid":30322,"timestamp":1531246021,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,628 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60760","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"2/0","dumpTime":1531246052}
2018-07-10T18:07:32,629 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60761 OPEN_TXN message :
\{"txnIds":null,"timestamp":1531246022,"fromTxnId":30323,"toTxnId":30323,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,653 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60761","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"3/0","dumpTime":1531246052}
2018-07-10T18:07:32,654 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60762 COMMIT_TXN message :
\{"txnid":30323,"timestamp":1531246028,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,679 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60762","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"4/0","dumpTime":1531246052}
2018-07-10T18:07:32,680 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60763 OPEN_TXN message :
\{"txnIds":null,"timestamp":1531246033,"fromTxnId":30324,"toTxnId":30324,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,702 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60763","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"5/0","dumpTime":1531246052}
2018-07-10T18:07:32,702 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60764 OPEN_TXN message :
\{"txnIds":null,"timestamp":1531246038,"fromTxnId":30325,"toTxnId":30325,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,724 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60764","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"6/0","dumpTime":1531246052}
2018-07-10T18:07:32,725 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60765 COMMIT_TXN message :
\{"txnid":30324,"timestamp":1531246038,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,747 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60765","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"7/0","dumpTime":1531246052}
2018-07-10T18:07:32,748 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60766 COMMIT_TXN message :
\{"txnid":30325,"timestamp":1531246043,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,768 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60766","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"8/0","dumpTime":1531246052}
2018-07-10T18:07:32,769 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60767 OPEN_TXN message :
\{"txnIds":null,"timestamp":1531246043,"fromTxnId":30326,"toTxnId":30326,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,791 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60767","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"9/0","dumpTime":1531246052}
2018-07-10T18:07:32,791 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60768 COMMIT_TXN message :
\{"txnid":30326,"timestamp":1531246049,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,816 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60768","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"10/0","dumpTime":1531246052}
2018-07-10T18:07:32,817 INFO [HiveServer2-Background-Pool: Thread-15161]:
events.AbstractEventHandler (:()) - Processing#60769 OPEN_TXN message :
\{"txnIds":null,"timestamp":1531246052,"fromTxnId":30327,"toTxnId":30327,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
2018-07-10T18:07:32,841 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::EVENT_DUMP:
\{"dbName":"*","eventId":"60769","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"11/0","dumpTime":1531246052}
2018-07-10T18:07:32,841 WARN [HiveServer2-Background-Pool: Thread-15161]:
metastore.RetryingMetaStoreClient (:()) - MetaStoreClient lost connection.
Attempting to reconnect (1 of 24) after 5s. getNextNotification
org.apache.thrift.transport.TTransportException: Cannot write to null
outputStream
at
org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:178)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:106)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:70)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_next_notification(ThriftHiveMetastore.java:5547)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_next_notification(ThriftHiveMetastore.java:5539)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getNextNotification(HiveMetaStoreClient.java:2697)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at com.sun.proxy.$Proxy60.getNextNotification(Unknown Source) ~[?:?]
at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2773)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at com.sun.proxy.$Proxy60.getNextNotification(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.metastore.messaging.EventUtils$MSClientNotificationFetcher.getNextNotificationEvents(EventUtils.java:94)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hadoop.hive.metastore.messaging.EventUtils$NotificationEventIterator.fetchNextBatch(EventUtils.java:146)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hadoop.hive.metastore.messaging.EventUtils$NotificationEventIterator.hasNext(EventUtils.java:176)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.incrementalDump(ReplDumpTask.java:170)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.execute(ReplDumpTask.java:121)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2668)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2339)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2015)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1713)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1707)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)
~[hive-service-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
~[hive-service-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)
~[hive-service-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
~[hadoop-common-3.1.0.3.0.0.0-1605.jar:?]
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:329)
~[hive-service-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_112]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_112]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[?:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
2018-07-10T18:07:33,157 INFO [org.apache.ranger.audit.queue.AuditBatchQueue1]:
destination.HDFSAuditDestination (:()) - Flushing HDFS audit. Event Size:4
2018-07-10T18:07:37,842 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient trying
reconnect as hive (auth:SIMPLE)
2018-07-10T18:07:37,842 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
thrift://hwx-demo-2.field.hortonworks.com:9083
2018-07-10T18:07:37,842 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore, current
connections: 19
2018-07-10T18:07:37,843 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Connected to metastore.
2018-07-10T18:07:37,864 INFO [HiveServer2-Background-Pool: Thread-15161]:
ReplState (:()) - REPL::END:
\{"dbName":"*","dumpType":"INCREMENTAL","actualNumEvents":11,"dumpEndTime":1531246057,"dumpDir":"/user/hive/repl/c72dc152-c252-4454-9b0e-cf8c335ab07c","lastReplId":"60769"}
2018-07-10T18:07:37,864 INFO [HiveServer2-Background-Pool: Thread-15161]:
repl.ReplDumpTask (:()) - Done dumping events, preparing to return
/user/hive/repl/c72dc152-c252-4454-9b0e-cf8c335ab07c,60769
2018-07-10T18:07:37,919 INFO [HiveServer2-Background-Pool: Thread-15161]:
ql.Driver (:()) - Completed executing
command(queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b); Time
taken: 5.419 seconds
2018-07-10T18:07:37,919 INFO [HiveServer2-Background-Pool: Thread-15161]:
ql.Driver (:()) - OK
2018-07-10T18:07:37,919 INFO [HiveServer2-Background-Pool: Thread-15161]:
lockmgr.DbTxnManager (:()) - Stopped heartbeat for query:
hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b
2018-07-10T18:07:37,921 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
thrift://hwx-demo-2.field.hortonworks.com:9083
2018-07-10T18:07:37,921 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore, current
connections: 20
2018-07-10T18:07:37,922 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.HiveMetaStoreClient (:()) - Connected to metastore.
2018-07-10T18:07:37,922 INFO [HiveServer2-Background-Pool: Thread-15161]:
metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient proxy=class
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive
(auth:SIMPLE) retries=24 delay=5 lifetime=0
2018-07-10T18:07:41,145 INFO [HiveServer2-Handler-Pool: Thread-14380]:
conf.HiveConf (HiveConf.java:getLogIdVar(5061)) - Using the default value
passed in for log id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:41,146 INFO [HiveServer2-Handler-Pool: Thread-14380]:
session.SessionState (:()) - Updating thread name to
f1e13736-3f10-4abf-a29b-683b534dfa4c HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:41,146 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: conf.HiveConf
(HiveConf.java:getLogIdVar(5061)) - Using the default value passed in for log
id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:41,146 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: session.SessionState (:()) - Resetting
thread name to HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:41,153 INFO [HiveServer2-Handler-Pool: Thread-14380]:
conf.HiveConf (HiveConf.java:getLogIdVar(5061)) - Using the default value
passed in for log id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:41,154 INFO [HiveServer2-Handler-Pool: Thread-14380]:
session.SessionState (:()) - Updating thread name to
f1e13736-3f10-4abf-a29b-683b534dfa4c HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:41,154 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: conf.HiveConf
(HiveConf.java:getLogIdVar(5061)) - Using the default value passed in for log
id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:41,155 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: session.SessionState (:()) - Resetting
thread name to HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:41,156 INFO [HiveServer2-Handler-Pool: Thread-14380]:
conf.HiveConf (HiveConf.java:getLogIdVar(5061)) - Using the default value
passed in for log id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:41,156 INFO [HiveServer2-Handler-Pool: Thread-14380]:
session.SessionState (:()) - Updating thread name to
f1e13736-3f10-4abf-a29b-683b534dfa4c HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:41,157 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: conf.HiveConf
(HiveConf.java:getLogIdVar(5061)) - Using the default value passed in for log
id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:41,157 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: session.SessionState (:()) - Resetting
thread name to HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:41,158 INFO [HiveServer2-Handler-Pool: Thread-14380]:
conf.HiveConf (HiveConf.java:getLogIdVar(5061)) - Using the default value
passed in for log id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:41,158 INFO [HiveServer2-Handler-Pool: Thread-14380]:
session.SessionState (:()) - Updating thread name to
f1e13736-3f10-4abf-a29b-683b534dfa4c HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:41,162 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: mapred.FileInputFormat (:()) - Total
input files to process : 1
2018-07-10T18:07:41,165 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: exec.ListSinkOperator (:()) -
RECORDS_OUT_INTERMEDIATE:0, RECORDS_OUT_OPERATOR_LIST_SINK_0:1,
2018-07-10T18:07:41,165 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: conf.HiveConf
(HiveConf.java:getLogIdVar(5061)) - Using the default value passed in for log
id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:41,166 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: session.SessionState (:()) - Resetting
thread name to HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:41,167 INFO [HiveServer2-Handler-Pool: Thread-14380]:
conf.HiveConf (HiveConf.java:getLogIdVar(5061)) - Using the default value
passed in for log id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:41,167 INFO [HiveServer2-Handler-Pool: Thread-14380]:
session.SessionState (:()) - Updating thread name to
f1e13736-3f10-4abf-a29b-683b534dfa4c HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:41,168 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: conf.HiveConf
(HiveConf.java:getLogIdVar(5061)) - Using the default value passed in for log
id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:41,168 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: session.SessionState (:()) - Resetting
thread name to HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:41,170 INFO [HiveServer2-Handler-Pool: Thread-14380]:
conf.HiveConf (HiveConf.java:getLogIdVar(5061)) - Using the default value
passed in for log id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:41,170 INFO [HiveServer2-Handler-Pool: Thread-14380]:
session.SessionState (:()) - Updating thread name to
f1e13736-3f10-4abf-a29b-683b534dfa4c HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:41,170 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: conf.HiveConf
(HiveConf.java:getLogIdVar(5061)) - Using the default value passed in for log
id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:41,171 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: session.SessionState (:()) - Resetting
thread name to HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:41,172 INFO [HiveServer2-Handler-Pool: Thread-14380]:
conf.HiveConf (HiveConf.java:getLogIdVar(5061)) - Using the default value
passed in for log id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:41,173 INFO [HiveServer2-Handler-Pool: Thread-14380]:
session.SessionState (:()) - Updating thread name to
f1e13736-3f10-4abf-a29b-683b534dfa4c HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:41,173 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: conf.HiveConf
(HiveConf.java:getLogIdVar(5061)) - Using the default value passed in for log
id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:41,173 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: session.SessionState (:()) - Resetting
thread name to HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:41,175 INFO [HiveServer2-Handler-Pool: Thread-14380]:
conf.HiveConf (HiveConf.java:getLogIdVar(5061)) - Using the default value
passed in for log id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:41,175 INFO [HiveServer2-Handler-Pool: Thread-14380]:
session.SessionState (:()) - Updating thread name to
f1e13736-3f10-4abf-a29b-683b534dfa4c HiveServer2-Handler-Pool: Thread-14380
2018-07-10T18:07:41,175 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: operation.OperationManager (:()) -
Closing operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=16eb1d07-e125-490c-8ab8-90192bfd459b]
2018-07-10T18:07:41,175 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: operation.OperationManager (:()) -
Removed queryId: hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b
corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=16eb1d07-e125-490c-8ab8-90192bfd459b]
2018-07-10T18:07:41,179 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: operation.Operation (:()) - Closing
operation log
/tmp/hive/operation_logs/f1e13736-3f10-4abf-a29b-683b534dfa4c/hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b
without delay
2018-07-10T18:07:41,179 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: conf.HiveConf
(HiveConf.java:getLogIdVar(5061)) - Using the default value passed in for log
id: f1e13736-3f10-4abf-a29b-683b534dfa4c
2018-07-10T18:07:41,180 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
HiveServer2-Handler-Pool: Thread-14380]: session.SessionState (:()) - Resetting
thread name to HiveServer2-Handler-Pool: Thread-14380
{code}
Metastore connection failed 1st attempt, but success after reconnect. That adds
5s for every repl dump command and likely to leak connection.
> Metastore connection fails for first attempt in repl dump.
> ----------------------------------------------------------
>
> Key: HIVE-20209
> URL: https://issues.apache.org/jira/browse/HIVE-20209
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
> Affects Versions: 3.0.0, 3.1.0, 4.0.0
> Reporter: Sankar Hariappan
> Assignee: Sankar Hariappan
> Priority: Major
> Labels: DR, replication
>
> Run the following command:
> {code:java}
> repl dump `*` from 60758 with ('hive.repl.dump.metadata.only'='true',
> 'hive.repl.dump.include.acid.tables'='true');
> {code}
> See this in hs2.log:
> {code:java}
> 2018-07-10T18:07:32,308 INFO [HiveServer2-Handler-Pool: Thread-14380]:
> conf.HiveConf (HiveConf.java:getLogIdVar(5061)) - Using the default value
> passed in for log id: f1e13736-3f10-4abf-a29b-683b534dfa4c
> 2018-07-10T18:07:32,309 INFO [HiveServer2-Handler-Pool: Thread-14380]:
> session.SessionState (:()) - Updating thread name to
> f1e13736-3f10-4abf-a29b-683b534dfa4c HiveServer2-Handler-Pool: Thread-14380
> 2018-07-10T18:07:32,311 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
> HiveServer2-Handler-Pool: Thread-14380]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=16eb1d07-e125-490c-8ab8-90192bfd459b]
> 2018-07-10T18:07:32,314 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
> HiveServer2-Handler-Pool: Thread-14380]: ql.Driver (:()) - Compiling
> command(queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b):
> repl dump `*` from 60758 with ('hive.repl.dump.metadata.only'='true',
> 'hive.repl.dump.include.acid.tables'='true')
> 2018-07-10T18:07:32,317 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
> HiveServer2-Handler-Pool: Thread-14380]: metastore.HiveMetaStoreClient (:())
> - Trying to connect to metastore with URI
> thrift://hwx-demo-2.field.hortonworks.com:9083
> 2018-07-10T18:07:32,317 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
> HiveServer2-Handler-Pool: Thread-14380]: metastore.HiveMetaStoreClient (:())
> - Opened a connection to metastore, current connections: 19
> 2018-07-10T18:07:32,319 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
> HiveServer2-Handler-Pool: Thread-14380]: metastore.HiveMetaStoreClient (:())
> - Connected to metastore.
> 2018-07-10T18:07:32,319 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
> HiveServer2-Handler-Pool: Thread-14380]: metastore.RetryingMetaStoreClient
> (:()) - RetryingMetaStoreClient proxy=class
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive
> (auth:SIMPLE) retries=24 delay=5 lifetime=0
> 2018-07-10T18:07:32,439 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
> HiveServer2-Handler-Pool: Thread-14380]: ql.Driver (:()) - Semantic Analysis
> Completed (retrial = false)
> 2018-07-10T18:07:32,440 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
> HiveServer2-Handler-Pool: Thread-14380]: ql.Driver (:()) - Returning Hive
> schema: Schema(fieldSchemas:[FieldSchema(name:dump_dir, type:string,
> comment:from deserializer), FieldSchema(name:last_repl_id, type:string,
> comment:from deserializer)], properties:null)
> 2018-07-10T18:07:32,443 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
> HiveServer2-Handler-Pool: Thread-14380]: exec.ListSinkOperator (:()) -
> Initializing operator LIST_SINK[0]
> 2018-07-10T18:07:32,446 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
> HiveServer2-Handler-Pool: Thread-14380]: ql.Driver (:()) - Completed
> compiling
> command(queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b);
> Time taken: 0.132 seconds
> 2018-07-10T18:07:32,447 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
> HiveServer2-Handler-Pool: Thread-14380]: conf.HiveConf
> (HiveConf.java:getLogIdVar(5061)) - Using the default value passed in for log
> id: f1e13736-3f10-4abf-a29b-683b534dfa4c
> 2018-07-10T18:07:32,448 INFO [f1e13736-3f10-4abf-a29b-683b534dfa4c
> HiveServer2-Handler-Pool: Thread-14380]: session.SessionState (:()) -
> Resetting thread name to HiveServer2-Handler-Pool: Thread-14380
> 2018-07-10T18:07:32,451 INFO [HiveServer2-Background-Pool: Thread-15161]:
> reexec.ReExecDriver (:()) - Execution #1 of query
> 2018-07-10T18:07:32,452 INFO [HiveServer2-Background-Pool: Thread-15161]:
> lockmgr.DbTxnManager (:()) - Setting lock request transaction to txnid:30327
> for queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b
> 2018-07-10T18:07:32,454 INFO [HiveServer2-Background-Pool: Thread-15161]:
> lockmgr.DbLockManager (:()) - Requesting:
> queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b
> LockRequest(component:[LockComponent(type:SHARED_READ, level:DB,
> dbname:default, operationType:SELECT), LockComponent(type:SHARED_READ,
> level:DB, dbname:hwxdemo, operationType:SELECT),
> LockComponent(type:SHARED_READ, level:DB, dbname:information_schema,
> operationType:SELECT), LockComponent(type:SHARED_READ, level:DB, dbname:sys,
> operationType:SELECT)], txnid:30327, user:hive,
> hostname:hwx-demo-2.field.hortonworks.com,
> agentInfo:hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b)
> 2018-07-10T18:07:32,497 INFO [HiveServer2-Background-Pool: Thread-15161]:
> lockmgr.DbLockManager (:()) - Response to
> queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b
> LockResponse(lockid:30305, state:ACQUIRED)
> 2018-07-10T18:07:32,501 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ql.Driver (:()) - Executing
> command(queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b):
> repl dump `*` from 60758 with ('hive.repl.dump.metadata.only'='true',
> 'hive.repl.dump.include.acid.tables'='true')
> 2018-07-10T18:07:32,503 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ql.Driver (:()) - Starting task [Stage-0:REPL_DUMP] in serial mode
> 2018-07-10T18:07:32,506 INFO [HiveServer2-Background-Pool: Thread-15161]:
> metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore,
> current connections: 18
> 2018-07-10T18:07:32,509 INFO [HiveServer2-Background-Pool: Thread-15161]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
> thrift://hwx-demo-2.field.hortonworks.com:9083
> 2018-07-10T18:07:32,509 INFO [HiveServer2-Background-Pool: Thread-15161]:
> metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore,
> current connections: 19
> 2018-07-10T18:07:32,510 INFO [HiveServer2-Background-Pool: Thread-15161]:
> metastore.HiveMetaStoreClient (:()) - Connected to metastore.
> 2018-07-10T18:07:32,510 INFO [HiveServer2-Background-Pool: Thread-15161]:
> metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient proxy=class
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive
> (auth:SIMPLE) retries=24 delay=5 lifetime=0
> 2018-07-10T18:07:32,536 INFO [HiveServer2-Background-Pool: Thread-15161]:
> metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore,
> current connections: 18
> 2018-07-10T18:07:32,538 INFO [HiveServer2-Background-Pool: Thread-15161]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
> thrift://hwx-demo-2.field.hortonworks.com:9083
> 2018-07-10T18:07:32,539 INFO [HiveServer2-Background-Pool: Thread-15161]:
> metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore,
> current connections: 19
> 2018-07-10T18:07:32,542 INFO [HiveServer2-Background-Pool: Thread-15161]:
> metastore.HiveMetaStoreClient (:()) - Connected to metastore.
> 2018-07-10T18:07:32,542 INFO [HiveServer2-Background-Pool: Thread-15161]:
> metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient proxy=class
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive
> (auth:SIMPLE) retries=24 delay=5 lifetime=0
> 2018-07-10T18:07:32,565 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ReplState (:()) - REPL::START:
> \{"dbName":"*","dumpType":"INCREMENTAL","estimatedNumEvents":0,"dumpStartTime":1531246052}
> 2018-07-10T18:07:32,577 INFO [HiveServer2-Background-Pool: Thread-15161]:
> metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore,
> current connections: 18
> 2018-07-10T18:07:32,578 INFO [HiveServer2-Background-Pool: Thread-15161]:
> events.AbstractEventHandler (:()) - Processing#60759 COMMIT_TXN message :
> \{"txnid":30321,"timestamp":1531246017,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
> 2018-07-10T18:07:32,603 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ReplState (:()) - REPL::EVENT_DUMP:
> \{"dbName":"*","eventId":"60759","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"1/0","dumpTime":1531246052}
> 2018-07-10T18:07:32,604 INFO [HiveServer2-Background-Pool: Thread-15161]:
> events.AbstractEventHandler (:()) - Processing#60760 COMMIT_TXN message :
> \{"txnid":30322,"timestamp":1531246021,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
> 2018-07-10T18:07:32,628 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ReplState (:()) - REPL::EVENT_DUMP:
> \{"dbName":"*","eventId":"60760","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"2/0","dumpTime":1531246052}
> 2018-07-10T18:07:32,629 INFO [HiveServer2-Background-Pool: Thread-15161]:
> events.AbstractEventHandler (:()) - Processing#60761 OPEN_TXN message :
> \{"txnIds":null,"timestamp":1531246022,"fromTxnId":30323,"toTxnId":30323,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
> 2018-07-10T18:07:32,653 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ReplState (:()) - REPL::EVENT_DUMP:
> \{"dbName":"*","eventId":"60761","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"3/0","dumpTime":1531246052}
> 2018-07-10T18:07:32,654 INFO [HiveServer2-Background-Pool: Thread-15161]:
> events.AbstractEventHandler (:()) - Processing#60762 COMMIT_TXN message :
> \{"txnid":30323,"timestamp":1531246028,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
> 2018-07-10T18:07:32,679 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ReplState (:()) - REPL::EVENT_DUMP:
> \{"dbName":"*","eventId":"60762","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"4/0","dumpTime":1531246052}
> 2018-07-10T18:07:32,680 INFO [HiveServer2-Background-Pool: Thread-15161]:
> events.AbstractEventHandler (:()) - Processing#60763 OPEN_TXN message :
> \{"txnIds":null,"timestamp":1531246033,"fromTxnId":30324,"toTxnId":30324,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
> 2018-07-10T18:07:32,702 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ReplState (:()) - REPL::EVENT_DUMP:
> \{"dbName":"*","eventId":"60763","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"5/0","dumpTime":1531246052}
> 2018-07-10T18:07:32,702 INFO [HiveServer2-Background-Pool: Thread-15161]:
> events.AbstractEventHandler (:()) - Processing#60764 OPEN_TXN message :
> \{"txnIds":null,"timestamp":1531246038,"fromTxnId":30325,"toTxnId":30325,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
> 2018-07-10T18:07:32,724 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ReplState (:()) - REPL::EVENT_DUMP:
> \{"dbName":"*","eventId":"60764","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"6/0","dumpTime":1531246052}
> 2018-07-10T18:07:32,725 INFO [HiveServer2-Background-Pool: Thread-15161]:
> events.AbstractEventHandler (:()) - Processing#60765 COMMIT_TXN message :
> \{"txnid":30324,"timestamp":1531246038,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
> 2018-07-10T18:07:32,747 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ReplState (:()) - REPL::EVENT_DUMP:
> \{"dbName":"*","eventId":"60765","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"7/0","dumpTime":1531246052}
> 2018-07-10T18:07:32,748 INFO [HiveServer2-Background-Pool: Thread-15161]:
> events.AbstractEventHandler (:()) - Processing#60766 COMMIT_TXN message :
> \{"txnid":30325,"timestamp":1531246043,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
> 2018-07-10T18:07:32,768 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ReplState (:()) - REPL::EVENT_DUMP:
> \{"dbName":"*","eventId":"60766","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"8/0","dumpTime":1531246052}
> 2018-07-10T18:07:32,769 INFO [HiveServer2-Background-Pool: Thread-15161]:
> events.AbstractEventHandler (:()) - Processing#60767 OPEN_TXN message :
> \{"txnIds":null,"timestamp":1531246043,"fromTxnId":30326,"toTxnId":30326,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
> 2018-07-10T18:07:32,791 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ReplState (:()) - REPL::EVENT_DUMP:
> \{"dbName":"*","eventId":"60767","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"9/0","dumpTime":1531246052}
> 2018-07-10T18:07:32,791 INFO [HiveServer2-Background-Pool: Thread-15161]:
> events.AbstractEventHandler (:()) - Processing#60768 COMMIT_TXN message :
> \{"txnid":30326,"timestamp":1531246049,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
> 2018-07-10T18:07:32,816 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ReplState (:()) - REPL::EVENT_DUMP:
> \{"dbName":"*","eventId":"60768","eventType":"EVENT_COMMIT_TXN","eventsDumpProgress":"10/0","dumpTime":1531246052}
> 2018-07-10T18:07:32,817 INFO [HiveServer2-Background-Pool: Thread-15161]:
> events.AbstractEventHandler (:()) - Processing#60769 OPEN_TXN message :
> \{"txnIds":null,"timestamp":1531246052,"fromTxnId":30327,"toTxnId":30327,"server":"thrift://hwx-demo-2.field.hortonworks.com:9083","servicePrincipal":"hive/[email protected]"}
> 2018-07-10T18:07:32,841 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ReplState (:()) - REPL::EVENT_DUMP:
> \{"dbName":"*","eventId":"60769","eventType":"EVENT_OPEN_TXN","eventsDumpProgress":"11/0","dumpTime":1531246052}
> 2018-07-10T18:07:32,841 WARN [HiveServer2-Background-Pool: Thread-15161]:
> metastore.RetryingMetaStoreClient (:()) - MetaStoreClient lost connection.
> Attempting to reconnect (1 of 24) after 5s. getNextNotification
> org.apache.thrift.transport.TTransportException: Cannot write to null
> outputStream
> at
> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at
> org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:178)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at
> org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:106)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:70)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_next_notification(ThriftHiveMetastore.java:5547)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_next_notification(ThriftHiveMetastore.java:5539)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getNextNotification(HiveMetaStoreClient.java:2697)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_112]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at com.sun.proxy.$Proxy60.getNextNotification(Unknown Source) ~[?:?]
> at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_112]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2773)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at com.sun.proxy.$Proxy60.getNextNotification(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.messaging.EventUtils$MSClientNotificationFetcher.getNextNotificationEvents(EventUtils.java:94)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at
> org.apache.hadoop.hive.metastore.messaging.EventUtils$NotificationEventIterator.fetchNextBatch(EventUtils.java:146)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at
> org.apache.hadoop.hive.metastore.messaging.EventUtils$NotificationEventIterator.hasNext(EventUtils.java:176)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.incrementalDump(ReplDumpTask.java:170)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.execute(ReplDumpTask.java:121)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2668)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2339)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2015)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1713)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1707)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
> ~[hive-exec-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)
> ~[hive-service-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
> ~[hive-service-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)
> ~[hive-service-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> ~[hadoop-common-3.1.0.3.0.0.0-1605.jar:?]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:329)
> ~[hive-service-3.1.0.3.0.0.0-1605.jar:3.1.0.3.0.0.0-1605]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_112]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_112]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[?:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> 2018-07-10T18:07:33,157 INFO
> [org.apache.ranger.audit.queue.AuditBatchQueue1]:
> destination.HDFSAuditDestination (:()) - Flushing HDFS audit. Event Size:4
> 2018-07-10T18:07:37,842 INFO [HiveServer2-Background-Pool: Thread-15161]:
> metastore.RetryingMetaStoreClient (:()) - RetryingMetaStoreClient trying
> reconnect as hive (auth:SIMPLE)
> 2018-07-10T18:07:37,842 INFO [HiveServer2-Background-Pool: Thread-15161]:
> metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI
> thrift://hwx-demo-2.field.hortonworks.com:9083
> 2018-07-10T18:07:37,842 INFO [HiveServer2-Background-Pool: Thread-15161]:
> metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore,
> current connections: 19
> 2018-07-10T18:07:37,843 INFO [HiveServer2-Background-Pool: Thread-15161]:
> metastore.HiveMetaStoreClient (:()) - Connected to metastore.
> 2018-07-10T18:07:37,864 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ReplState (:()) - REPL::END:
> \{"dbName":"*","dumpType":"INCREMENTAL","actualNumEvents":11,"dumpEndTime":1531246057,"dumpDir":"/user/hive/repl/c72dc152-c252-4454-9b0e-cf8c335ab07c","lastReplId":"60769"}
> 2018-07-10T18:07:37,864 INFO [HiveServer2-Background-Pool: Thread-15161]:
> repl.ReplDumpTask (:()) - Done dumping events, preparing to return
> /user/hive/repl/c72dc152-c252-4454-9b0e-cf8c335ab07c,60769
> 2018-07-10T18:07:37,919 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ql.Driver (:()) - Completed executing
> command(queryId=hive_20180710180732_7dcc20db-90db-486d-a825-e6fa91dc092b);
> Time taken: 5.419 seconds
> 2018-07-10T18:07:37,919 INFO [HiveServer2-Background-Pool: Thread-15161]:
> ql.Driver (:()) - OK
> {code}
> Metastore connection failed 1st attempt, but success after reconnect. That
> adds 5s for every repl dump command and likely to leak connection.
> Similarly, Hive.close() also causes
> {code:java}
> org.apache.thrift.transport.TTransportException: Cannot write to null
> outputStream{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
