[
https://issues.apache.org/jira/browse/FLUME-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16442177#comment-16442177
]
ASF GitHub Bot commented on FLUME-2956:
---------------------------------------
GitHub user hejiang2000 opened a pull request:
https://github.com/apache/flume/pull/206
Fix the bug FLUME-2956
1. flush all writers to make sure heartbeats are sent on inactive ones
2. catch any throwable in abortAllWriters and closeAllWriters to guarantee
nothing left in allWriters map
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hejiang2000/flume hive-sink-FLUME-2956
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flume/pull/206.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #206
----
commit 8e49b7612290cb625f85dd495f787afac2524cce
Author: [email protected] <hejiang@...>
Date: 2018-04-18T09:15:40Z
Fix the bug FLUME-2956
----
> hive sink not sending heartbeat correctly
> -----------------------------------------
>
> Key: FLUME-2956
> URL: https://issues.apache.org/jira/browse/FLUME-2956
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: 1.5.2
> Environment: linux CentOS 6.6
> Reporter: Olivier brobecker
> Priority: Major
> Attachments: flume-hive-sink-1.8.0.jar, hive-sink-patch.txt
>
>
> Flume has been configured in order to populate a hive table as follow :
> agentSCDR.sinks.hive1.type = hive
> agentSCDR.sinks.hive1.channel = channel1
> agentSCDR.sinks.hive1.hive.metastore = thrift://myserver:9083
> agentSCDR.sinks.hive1.hive.txnsPerBatchAsk = 10
> agentSCDR.sinks.hive1.hive.database = myDatabase
> agentSCDR.sinks.hive1.hive.table = my_table
> agentSCDR.sinks.hive1.hive.partition = %Y
> agentSCDR.sinks.hive1.heartBeatInterval = 60
> agentSCDR.sinks.hive1.serializer = DELIMITED
> agentSCDR.sinks.hive1.serializer.delimiter = ;
> agentSCDR.sinks.hive1.serializer.serdeSeparator = ;
> agentSCDR.sinks.hive1.serializer.fieldnames = field1,field2,field3...
> My dataflow is irregular at best and I can have more than 30 minutes of
> inactivity, so I have a heartbeatInterval at 60s in order to keep my txn
> alive.
> The issue is that this heartBeat is only sent when flume is trying to inject
> some datas into hive instead of having one every 60s.
> # grep -i heartbeat flume-agentSCDR.log
> 15 juil. 2016 13:40:43,008 INFO [hive-hive1-call-runner-0]
> (org.apache.flume.sink.hive.HiveWriter$2.call:238) - Sending heartbeat on
> batch TxnIds=[3755...3764] on endPoint = {metaStoreUri=...
> 15 juil. 2016 14:12:21,001 INFO [hive-hive1-call-runner-0]
> (org.apache.flume.sink.hive.HiveWriter$2.call:231) - Sending heartbeat on
> batch TxnIds=[3785...3794] on endPoint = {metaStoreUri=...
> 15 juil. 2016 14:27:56,963 INFO [hive-hive1-call-runner-0]
> (org.apache.flume.sink.hive.HiveWriter$2.call:231) - Sending heartbeat on
> batch TxnIds=[3795...3804] on endPoint = {metaStoreUri=...
> ...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
