Olivier brobecker created FLUME-2956:
----------------------------------------
Summary: hive sink not sending heartbeat correctly
Key: FLUME-2956
URL: https://issues.apache.org/jira/browse/FLUME-2956
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: v1.5.2
Environment: linux CentOS 6.6
Reporter: Olivier brobecker
Flume has been configured in order to populate a hive table as follow :
agentSCDR.sinks.hive1.type = hive
agentSCDR.sinks.hive1.channel = channel1
agentSCDR.sinks.hive1.hive.metastore = thrift://myserver:9083
agentSCDR.sinks.hive1.hive.txnsPerBatchAsk = 10
agentSCDR.sinks.hive1.hive.database = myDatabase
agentSCDR.sinks.hive1.hive.table = my_table
agentSCDR.sinks.hive1.hive.partition = %Y
agentSCDR.sinks.hive1.heartBeatInterval = 60
agentSCDR.sinks.hive1.serializer = DELIMITED
agentSCDR.sinks.hive1.serializer.delimiter = ;
agentSCDR.sinks.hive1.serializer.serdeSeparator = ;
agentSCDR.sinks.hive1.serializer.fieldnames = field1,field2,field3...
My dataflow is irregular at best and I can have more than 30 minutes of
inactivity, so I have a heartbeatInterval at 60s in order to keep my txn alive.
The issue is that this heartBeat is only sent when flume is trying to inject
some datas into hive instead of having one every 60s.
# grep -i heartbeat flume-agentSCDR.log
15 juil. 2016 13:40:43,008 INFO [hive-hive1-call-runner-0]
(org.apache.flume.sink.hive.HiveWriter$2.call:238) - Sending heartbeat on
batch TxnIds=[3755...3764] on endPoint = {metaStoreUri=...
15 juil. 2016 14:12:21,001 INFO [hive-hive1-call-runner-0]
(org.apache.flume.sink.hive.HiveWriter$2.call:231) - Sending heartbeat on
batch TxnIds=[3785...3794] on endPoint = {metaStoreUri=...
15 juil. 2016 14:27:56,963 INFO [hive-hive1-call-runner-0]
(org.apache.flume.sink.hive.HiveWriter$2.call:231) - Sending heartbeat on
batch TxnIds=[3795...3804] on endPoint = {metaStoreUri=...
...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
