[
https://issues.apache.org/jira/browse/FLUME-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292121#comment-16292121
]
Abhishek Choudhary commented on FLUME-2956:
-------------------------------------------
Hi
I didn't find any functional issue with the patch yet. Just one issue, just one
improvement in term of logging.
If log level is set to INFO, then log files are filled with "Sending hearbeat
for" messages for every second when drainOneBatch is called.
It should be part of DEBUG flag for all inactive writers.
> hive sink not sending heartbeat correctly
> -----------------------------------------
>
> Key: FLUME-2956
> URL: https://issues.apache.org/jira/browse/FLUME-2956
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: 1.5.2
> Environment: linux CentOS 6.6
> Reporter: Olivier brobecker
> Attachments: flume-hive-sink-1.8.0.jar, hive-sink-patch.txt
>
>
> Flume has been configured in order to populate a hive table as follow :
> agentSCDR.sinks.hive1.type = hive
> agentSCDR.sinks.hive1.channel = channel1
> agentSCDR.sinks.hive1.hive.metastore = thrift://myserver:9083
> agentSCDR.sinks.hive1.hive.txnsPerBatchAsk = 10
> agentSCDR.sinks.hive1.hive.database = myDatabase
> agentSCDR.sinks.hive1.hive.table = my_table
> agentSCDR.sinks.hive1.hive.partition = %Y
> agentSCDR.sinks.hive1.heartBeatInterval = 60
> agentSCDR.sinks.hive1.serializer = DELIMITED
> agentSCDR.sinks.hive1.serializer.delimiter = ;
> agentSCDR.sinks.hive1.serializer.serdeSeparator = ;
> agentSCDR.sinks.hive1.serializer.fieldnames = field1,field2,field3...
> My dataflow is irregular at best and I can have more than 30 minutes of
> inactivity, so I have a heartbeatInterval at 60s in order to keep my txn
> alive.
> The issue is that this heartBeat is only sent when flume is trying to inject
> some datas into hive instead of having one every 60s.
> # grep -i heartbeat flume-agentSCDR.log
> 15 juil. 2016 13:40:43,008 INFO [hive-hive1-call-runner-0]
> (org.apache.flume.sink.hive.HiveWriter$2.call:238) - Sending heartbeat on
> batch TxnIds=[3755...3764] on endPoint = {metaStoreUri=...
> 15 juil. 2016 14:12:21,001 INFO [hive-hive1-call-runner-0]
> (org.apache.flume.sink.hive.HiveWriter$2.call:231) - Sending heartbeat on
> batch TxnIds=[3785...3794] on endPoint = {metaStoreUri=...
> 15 juil. 2016 14:27:56,963 INFO [hive-hive1-call-runner-0]
> (org.apache.flume.sink.hive.HiveWriter$2.call:231) - Sending heartbeat on
> batch TxnIds=[3795...3804] on endPoint = {metaStoreUri=...
> ...
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
