Hi,
I am using flume 1.7.0 on AWS EMR cluster with hive version 2.1.1. Flume is
configured for hive-sink to write data to hive metastore using hive
streaming. flume hive-sink commit transactions for few minutes but after
some time (10-15 minutes), transactions aborted and sink runner stopped. In
order to restart the hive sink runner again, I need to reload the flume.
2017-11-08 15:38:44,826 (hive-sink3-call-runner-0) [INFO -
org.apache.flume.sink.hive.HiveWriter$2.call(HiveWriter.java:236)] Sending
heartbeat on batch TxnIds=[1491330
...1491339] on endPoint = {metaStoreUri='thrift://localhost:9083',
database='bsfc', table='rawdata', partitionVals=[201711990] }
2017-11-08 15:38:44,830 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[WARN -
org.apache.flume.sink.hive.HiveWriter.heartBeat(HiveWriter.java:244)]
Unable to send hea
rtbeat on Txn Batch TxnIds=[1491330...1491339] on endPoint =
{metaStoreUri='thrift://localhost:9083', database='bsfc', table='rawdata',
partitionVals=[201711990] }
org.apache.hive.hcatalog.streaming.HeartBeatFailure: Heart beat error.
InvalidTxns: []. AbortedTxns: [1491334, 1491335, 1491338, 1491339, 1491336,
1491337]
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.heartbeat(HiveEndPoint.java:953)
at org.apache.flume.sink.hive.HiveWriter$2.call(HiveWriter.java:237)
at org.apache.flume.sink.hive.HiveWriter$2.call(HiveWriter.java:233)
at
org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2017-11-08 15:38:44,830 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[INFO -
org.apache.flume.sink.hive.HiveWriter.commitTxn(HiveWriter.java:337)]
Committing Txn 149
1334 on EndPoint: {metaStoreUri='thrift://localhost:9083', database='bsfc',
table='rawdata', partitionVals=[201711990] }
2017-11-08 15:38:44,833 (hive-sink3-call-runner-0) [ERROR -
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.markDead(HiveEndPoint.java:756)]
Fatal
error on TxnIds=[1491330...1491339] on endPoint =
{metaStoreUri='thrift://localhost:9083', database='bsfc', table='rawdata',
partitionVals=[201711990] }; cause Unable t
o abort invalid transaction id : 1491334: No such transaction txnid:1491334
org.apache.hive.hcatalog.streaming.TransactionError: Unable to abort
invalid transaction id : 1491334: No such transaction txnid:1491334
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abortImpl(HiveEndPoint.java:936)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abort(HiveEndPoint.java:894)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.markDead(HiveEndPoint.java:753)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.commit(HiveEndPoint.java:853)
at org.apache.flume.sink.hive.HiveWriter$6.call(HiveWriter.java:343)
at org.apache.flume.sink.hive.HiveWriter$6.call(HiveWriter.java:340)
at
org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: NoSuchTxnException(message:No such transaction txnid:1491334)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result$abort_txn_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result$abort_txn_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result.read(ThriftHiveMetastore.java)
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_abort_txn(ThriftHiveMetastore.java:4477)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.abort_txn(ThriftHiveMetastore.java:4464)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.rollbackTxn(HiveMetaStoreClient.java:2093)
at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
at com.sun.proxy.$Proxy8.rollbackTxn(Unknown Source)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abortImpl(HiveEndPoint.java:923)
... 10 more
Attached is the flume error file
Thanks & Regards,
Amit Kumar,
Mob: 9910611621
2017-11-08 15:38:44,826 (hive-sink3-call-runner-0) [INFO -
org.apache.flume.sink.hive.HiveWriter$2.call(HiveWriter.java:236)] Sending
heartbeat on batch TxnIds=[1491330
...1491339] on endPoint = {metaStoreUri='thrift://localhost:9083',
database='bsfc', table='rawdata', partitionVals=[201711990] }
2017-11-08 15:38:44,830 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN -
org.apache.flume.sink.hive.HiveWriter.heartBeat(HiveWriter.java:244)] Unable to
send hea
rtbeat on Txn Batch TxnIds=[1491330...1491339] on endPoint =
{metaStoreUri='thrift://localhost:9083', database='bsfc', table='rawdata',
partitionVals=[201711990] }
org.apache.hive.hcatalog.streaming.HeartBeatFailure: Heart beat error.
InvalidTxns: []. AbortedTxns: [1491334, 1491335, 1491338, 1491339, 1491336,
1491337]
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.heartbeat(HiveEndPoint.java:953)
at org.apache.flume.sink.hive.HiveWriter$2.call(HiveWriter.java:237)
at org.apache.flume.sink.hive.HiveWriter$2.call(HiveWriter.java:233)
at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2017-11-08 15:38:44,830 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hive.HiveWriter.commitTxn(HiveWriter.java:337)]
Committing Txn 149
1334 on EndPoint: {metaStoreUri='thrift://localhost:9083', database='bsfc',
table='rawdata', partitionVals=[201711990] }
2017-11-08 15:38:44,833 (hive-sink3-call-runner-0) [ERROR -
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.markDead(HiveEndPoint.java:756)]
Fatal
error on TxnIds=[1491330...1491339] on endPoint =
{metaStoreUri='thrift://localhost:9083', database='bsfc', table='rawdata',
partitionVals=[201711990] }; cause Unable t
o abort invalid transaction id : 1491334: No such transaction txnid:1491334
org.apache.hive.hcatalog.streaming.TransactionError: Unable to abort invalid
transaction id : 1491334: No such transaction txnid:1491334
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abortImpl(HiveEndPoint.java:936)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abort(HiveEndPoint.java:894)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.markDead(HiveEndPoint.java:753)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.commit(HiveEndPoint.java:853)
at org.apache.flume.sink.hive.HiveWriter$6.call(HiveWriter.java:343)
at org.apache.flume.sink.hive.HiveWriter$6.call(HiveWriter.java:340)
at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: NoSuchTxnException(message:No such transaction txnid:1491334)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result$abort_txn_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result$abort_txn_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_abort_txn(ThriftHiveMetastore.java:4477)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.abort_txn(ThriftHiveMetastore.java:4464)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.rollbackTxn(HiveMetaStoreClient.java:2093)
at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
at com.sun.proxy.$Proxy8.rollbackTxn(Unknown Source)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abortImpl(HiveEndPoint.java:923)
... 10 more
2017-11-08 15:38:44,834 (hive-sink3-call-runner-0) [INFO -
com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.close(MultipartUploadOutputStream.java:384)]
clos
e closed:false
s3://bsfcrawdata/partcol=201711990/delta_1491330_1491339/bucket_00000
2017-11-08 15:38:44,958 (hive-sink3-call-runner-0) [INFO -
com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.close(MultipartUploadOutputStream.java:384)]
clos
e closed:false
s3://bsfcrawdata/partcol=201711990/delta_1491330_1491339/bucket_00000_flush_length
2017-11-08 15:38:45,056 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN -
org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:323)] sink3 :
Commit of
Txn 1491334 failed on EndPoint: {metaStoreUri='thrift://localhost:9083',
database='bsfc', table='rawdata', partitionVals=[201711990] }
org.apache.flume.sink.hive.HiveWriter$CommitException: Commit of Txn 1491334
failed on EndPoint: {metaStoreUri='thrift://localhost:9083', database='bsfc',
table='rawdat
a', partitionVals=[201711990] }
at org.apache.flume.sink.hive.HiveWriter.commitTxn(HiveWriter.java:348)
at org.apache.flume.sink.hive.HiveWriter.flush(HiveWriter.java:199)
at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:316)
at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:253)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Aborted
transaction cannot be committed: Transaction txnid:1491334 already aborted
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.commitImpl(HiveEndPoint.java:866)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.commit(HiveEndPoint.java:834)
at org.apache.flume.sink.hive.HiveWriter$6.call(HiveWriter.java:343)
at org.apache.flume.sink.hive.HiveWriter$6.call(HiveWriter.java:340)
at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: TxnAbortedException(message:Transaction txnid:1491334 already
aborted)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$commit_txn_result$commit_txn_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$commit_txn_result$commit_txn_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$commit_txn_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_commit_txn(ThriftHiveMetastore.java:4523)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.commit_txn(ThriftHiveMetastore.java:4510)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.commitTxn(HiveMetaStoreClient.java:2099)
at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
at com.sun.proxy.$Proxy8.commitTxn(Unknown Source)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.commitImpl(HiveEndPoint.java:860)
... 8 more
2017-11-08 15:38:45,056 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hive.HiveWriter.abortTxn(HiveWriter.java:353)] Aborting
Txn id 149
1349 on End Point {metaStoreUri='thrift://localhost:9083', database='bsfc',
table='rawdata', partitionVals=[201711976] }
2017-11-08 15:38:45,060 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hive.HiveWriter.abortTxn(HiveWriter.java:353)] Aborting
Txn id 149
1370 on End Point {metaStoreUri='thrift://localhost:9083', database='bsfc',
table='rawdata', partitionVals=[201711982] }
2017-11-08 15:38:45,062 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hive.HiveWriter.abortTxn(HiveWriter.java:353)] Aborting
Txn id 149
1329 on End Point {metaStoreUri='thrift://localhost:9083', database='bsfc',
table='rawdata', partitionVals=[201711983] }
2017-11-08 15:38:45,064 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hive.HiveWriter.abortTxn(HiveWriter.java:353)] Aborting
Txn id 149
1262 on End Point {metaStoreUri='thrift://localhost:9083', database='bsfc',
table='rawdata', partitionVals=[201711981] }
2017-11-08 15:38:45,065 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN -
org.apache.flume.sink.hive.HiveWriter.abortTxn(HiveWriter.java:368)] Error
aborting Txn
1491262 on EndPoint: {metaStoreUri='thrift://localhost:9083', database='bsfc',
table='rawdata', partitionVals=[201711981] }
org.apache.hive.hcatalog.streaming.TransactionError: Unable to abort invalid
transaction id : 1491262: No such transaction txnid:1491262
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abortImpl(HiveEndPoint.java:936)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abort(HiveEndPoint.java:894)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abort(HiveEndPoint.java:889)
at org.apache.flume.sink.hive.HiveWriter$7.call(HiveWriter.java:358)
at org.apache.flume.sink.hive.HiveWriter$7.call(HiveWriter.java:355)
at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: NoSuchTxnException(message:No such transaction txnid:1491262)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result$abort_txn_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result$abort_txn_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_abort_txn(ThriftHiveMetastore.java:4477)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.abort_txn(ThriftHiveMetastore.java:4464)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.rollbackTxn(HiveMetaStoreClient.java:2093)
at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
at com.sun.proxy.$Proxy8.rollbackTxn(Unknown Source)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abortImpl(HiveEndPoint.java:929)
... 9 more
2017-11-08 15:38:45,065 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hive.HiveWriter.abortTxn(HiveWriter.java:353)] Aborting
Txn id 149
1334 on End Point {metaStoreUri='thrift://localhost:9083', database='bsfc',
table='rawdata', partitionVals=[201711990] }
2017-11-08 15:38:45,066 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hive.HiveWriter.closeTxnBatch(HiveWriter.java:407)]
Closing Txn Ba
tch TxnIds=[1491340...1491349] on endPoint =
{metaStoreUri='thrift://localhost:9083', database='bsfc', table='rawdata',
partitionVals=[201711976] }.
2017-11-08 15:38:45,066 (hive-sink3-call-runner-0) [INFO -
com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.close(MultipartUploadOutputStream.java:384)]
clos
e closed:false
s3://bsfcrawdata/partcol=201711976/delta_1491340_1491349/bucket_00000
2017-11-08 15:38:45,090 (hive-sink3-call-runner-0) [INFO -
com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.close(MultipartUploadOutputStream.java:384)]
clos
e closed:false
s3://bsfcrawdata/partcol=201711976/delta_1491340_1491349/bucket_00000_flush_length
2017-11-08 15:38:45,175 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hive.HiveWriter.closeConnection(HiveWriter.java:319)]
Closing conn
ection to EndPoint : {metaStoreUri='thrift://localhost:9083', database='bsfc',
table='rawdata', partitionVals=[201711976] }
2017-11-08 15:38:45,186 (hive-sink3-call-runner-0) [INFO -
org.apache.flume.sink.hive.HiveWriter$4.call(HiveWriter.java:297)] Aborted txn
1491371
2017-11-08 15:38:45,198 (hive-sink3-call-runner-0) [INFO -
org.apache.flume.sink.hive.HiveWriter$4.call(HiveWriter.java:297)] Aborted txn
1491372
2017-11-08 15:38:45,207 (hive-sink3-call-runner-0) [INFO -
org.apache.flume.sink.hive.HiveWriter$4.call(HiveWriter.java:297)] Aborted txn
1491373
2017-11-08 15:38:45,215 (hive-sink3-call-runner-0) [INFO -
org.apache.flume.sink.hive.HiveWriter$4.call(HiveWriter.java:297)] Aborted txn
1491374
2017-11-08 15:38:45,221 (hive-sink3-call-runner-0) [INFO -
org.apache.flume.sink.hive.HiveWriter$4.call(HiveWriter.java:297)] Aborted txn
1491375
2017-11-08 15:38:45,227 (hive-sink3-call-runner-0) [INFO -
org.apache.flume.sink.hive.HiveWriter$4.call(HiveWriter.java:297)] Aborted txn
1491376
2017-11-08 15:38:45,233 (hive-sink3-call-runner-0) [INFO -
org.apache.flume.sink.hive.HiveWriter$4.call(HiveWriter.java:297)] Aborted txn
1491377
2017-11-08 15:38:45,239 (hive-sink3-call-runner-0) [INFO -
org.apache.flume.sink.hive.HiveWriter$4.call(HiveWriter.java:297)] Aborted txn
1491378
2017-11-08 15:38:45,245 (hive-sink3-call-runner-0) [INFO -
org.apache.flume.sink.hive.HiveWriter$4.call(HiveWriter.java:297)] Aborted txn
1491379
2017-11-08 15:38:45,245 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hive.HiveWriter.closeTxnBatch(HiveWriter.java:407)]
Closing Txn Ba
tch TxnIds=[1491370...1491379] on endPoint =
{metaStoreUri='thrift://localhost:9083', database='bsfc', table='rawdata',
partitionVals=[201711982] }.
2017-11-08 15:38:45,314 (hive-sink3-call-runner-0) [INFO -
com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.close(MultipartUploadOutputStream.java:384)]
clos
e closed:false
s3://bsfcrawdata/partcol=201711982/delta_1491370_1491379/bucket_00000
2017-11-08 15:38:45,359 (hive-sink3-call-runner-0) [INFO -
com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.close(MultipartUploadOutputStream.java:384)]
clos
e closed:false
s3://bsfcrawdata/partcol=201711982/delta_1491370_1491379/bucket_00000_flush_length
2017-11-08 15:38:45,449 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hive.HiveWriter.closeConnection(HiveWriter.java:319)]
Closing conn
ection to EndPoint : {metaStoreUri='thrift://localhost:9083', database='bsfc',
table='rawdata', partitionVals=[201711982] }
2017-11-08 15:38:45,449 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hive.HiveWriter.closeTxnBatch(HiveWriter.java:407)]
Closing Txn Ba
tch TxnIds=[1491320...1491329] on endPoint =
{metaStoreUri='thrift://localhost:9083', database='bsfc', table='rawdata',
partitionVals=[201711983] }.
2017-11-08 15:38:45,449 (hive-sink3-call-runner-0) [INFO -
com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.close(MultipartUploadOutputStream.java:384)]
clos
e closed:false
s3://bsfcrawdata/partcol=201711983/delta_1491320_1491329/bucket_00000
2017-11-08 15:38:45,468 (hive-sink3-call-runner-0) [INFO -
com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.close(MultipartUploadOutputStream.java:384)]
clos
e closed:false
s3://bsfcrawdata/partcol=201711983/delta_1491320_1491329/bucket_00000_flush_length
2017-11-08 15:38:45,556 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hive.HiveWriter.closeConnection(HiveWriter.java:319)]
Closing conn
ection to EndPoint : {metaStoreUri='thrift://localhost:9083', database='bsfc',
table='rawdata', partitionVals=[201711983] }
2017-11-08 15:38:45,557 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN -
org.apache.flume.sink.hive.HiveWriter.abortCurrTxnHelper(HiveWriter.java:303)]
Unable to
abort transaction 1491262
org.apache.hive.hcatalog.streaming.TransactionError: Unable to abort invalid
transaction id : 1491262: No such transaction txnid:1491262
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abortImpl(HiveEndPoint.java:936)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abort(HiveEndPoint.java:894)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abort(HiveEndPoint.java:889)
at org.apache.flume.sink.hive.HiveWriter$4.call(HiveWriter.java:296)
at org.apache.flume.sink.hive.HiveWriter$4.call(HiveWriter.java:293)
at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: NoSuchTxnException(message:No such transaction txnid:1491262)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result$abort_txn_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result$abort_txn_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_abort_txn(ThriftHiveMetastore.java:4477)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.abort_txn(ThriftHiveMetastore.java:4464)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.rollbackTxn(HiveMetaStoreClient.java:2093)
at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
at com.sun.proxy.$Proxy8.rollbackTxn(Unknown Source)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abortImpl(HiveEndPoint.java:929)
... 9 more
2017-11-08 15:38:45,561 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN -
org.apache.flume.sink.hive.HiveWriter.abortRemainingTxns(HiveWriter.java:282)]
Error whe
n aborting remaining transactions in batch TxnIds=[1491260...1491269] on
endPoint = {metaStoreUri='thrift://localhost:9083', database='bsfc',
table='rawdata', partition
Vals=[201711981] }
org.apache.hive.hcatalog.streaming.TransactionError: Unable to acquire lock on
{metaStoreUri='thrift://localhost:9083', database='bsfc', table='rawdata',
partitionVals=
[201711981] }: Transaction txnid:1491263 already aborted
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:685)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:652)
at org.apache.flume.sink.hive.HiveWriter$3.call(HiveWriter.java:275)
at org.apache.flume.sink.hive.HiveWriter$3.call(HiveWriter.java:272)
at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: TxnAbortedException(message:Transaction txnid:1491263 already
aborted)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$lock_result$lock_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$lock_result$lock_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$lock_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:4549)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:4536)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:2115)
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
at com.sun.proxy.$Proxy8.lock(Unknown Source)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:680)
... 8 more
2017-11-08 15:38:45,561 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hive.HiveWriter.closeTxnBatch(HiveWriter.java:407)]
Closing Txn Ba
tch TxnIds=[1491260...1491269] on endPoint =
{metaStoreUri='thrift://localhost:9083', database='bsfc', table='rawdata',
partitionVals=[201711981] }.
2017-11-08 15:38:45,562 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN -
org.apache.flume.sink.hive.HiveWriter.closeTxnBatch(HiveWriter.java:418)] Error
closing
Txn Batch TxnIds=[1491260...1491269] on endPoint =
{metaStoreUri='thrift://localhost:9083', database='bsfc', table='rawdata',
partitionVals=[201711981] }
org.apache.hive.hcatalog.streaming.TransactionError: Unable to abort invalid
transaction id : 1491263: No such transaction txnid:1491263
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abortImpl(HiveEndPoint.java:936)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.close(HiveEndPoint.java:975)
at org.apache.flume.sink.hive.HiveWriter$10.call(HiveWriter.java:411)
at org.apache.flume.sink.hive.HiveWriter$10.call(HiveWriter.java:408)
at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: NoSuchTxnException(message:No such transaction txnid:1491263)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result$abort_txn_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result$abort_txn_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$abort_txn_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_abort_txn(ThriftHiveMetastore.java:4477)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.abort_txn(ThriftHiveMetastore.java:4464)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.rollbackTxn(HiveMetaStoreClient.java:2093)
at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
at com.sun.proxy.$Proxy8.rollbackTxn(Unknown Source)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.abortImpl(HiveEndPoint.java:923)
... 8 more
2017-11-08 15:38:45,563 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hive.HiveWriter.closeConnection(HiveWriter.java:319)]
Closing conn
ection to EndPoint : {metaStoreUri='thrift://localhost:9083', database='bsfc',
table='rawdata', partitionVals=[201711981] }
2017-11-08 15:38:45,563 (hive-sink3-call-runner-0) [INFO -
org.apache.flume.sink.hive.HiveWriter$4.call(HiveWriter.java:297)] Aborted txn
1491334
2017-11-08 15:38:45,563 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR
- org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to
deliver e
vent. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.IllegalStateException:
TransactionBatch TxnIds=[1491330...1491339] on endPoint =
{metaStoreUri='thrift://localhost:90
83', database='bsfc', table='rawdata', partitionVals=[201711990] } has been
closed()
at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:267)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: TransactionBatch
TxnIds=[1491330...1491339] on endPoint =
{metaStoreUri='thrift://localhost:9083', database='bsfc', table='r
awdata', partitionVals=[201711990] } has been closed()
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.checkIsClosed(HiveEndPoint.java:739)
at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:650)
at org.apache.flume.sink.hive.HiveWriter$3.call(HiveWriter.java:275)
at org.apache.flume.sink.hive.HiveWriter$3.call(HiveWriter.java:272)
at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
