[
https://issues.apache.org/jira/browse/FLUME-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15518810#comment-15518810
]
hailin edited comment on FLUME-2996 at 9/24/16 10:14 AM:
---------------------------------------------------------
hi bellow is the configure file content:
a1.sources = r1
a1.sinks = sink_etl_product sink_kinesis_stream
a1.channels = channel_etl_product channel_kinesis_stream
#source setting
#a1.sources.r1.type = http
a1.sources.r1.type = flume.source.HTTPSource
a1.sources.r1.port = 6558
a1.sources.r1.dataSize = 1
a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = event_type
a1.sources.r1.selector.mapping.etl_product = channel_etl_product
channel_kinesis_stream
#handler setting
a1.sources.r1.handler = flume.source.BlobExtHandler
a1.sources.r1.handler.gzip = true
a1.sources.r1.handler.maxBlobLength = 100000
a1.channels.channel_etl_product.type = file
a1.channels.channel_etl_product.checkpointDir = ./checkpoint/product
a1.channels.channel_etl_product.dataDirs = ./data/product
a1.channels.channel_etl_product.checkpointInterval = 10
#Kinesis Firehose
a1.sinks.sink_etl_mix.type = flume.sink.KinesisFirehoseSink
a1.sinks.sink_etl_mix.channel = channel_etl_product
a1.sinks.sink_etl_mix.streamName = product_flumeFirehoseStream
a1.sinks.sink_etl_mix.endpoint = {firehose_endpoint}
flume.source.HTTPSource,flume.source.BlobExtHandler,flume.sink.KinesisFirehoseSink,
these 3 Java class contains our bussiness logic.
what I want is that the flume will transfer data into AWS Kinesis Firehose.
The application works fine on AWS EC2 linux instance. What I meet is flume
shutdown automaticly after several hours , the process disappear when run "ps
-ef | grep flume".
I check the flume log, no errors found, seems it shutdown normally:
24 Sep 2016 05:53:59,391 INFO [agent-shutdown-hook]
(org.apache.flume.lifecycle.LifecycleSupervisor.stop:79) - Stopping lifecycle
supervisor 10
24 Sep 2016 05:53:59,394 INFO [agent-shutdown-hook]
(org.apache.flume.channel.file.FileChannel.stop:321) - Stopping FileChannel
channel_etl_product { dataDirs: [./data/product] }...
24 Sep 2016 05:53:59,395 INFO [agent-shutdown-hook]
(org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint:230)
- Start checkpoint for ./checkpoint/product/checkpoint, elements to sync = 0
24 Sep 2016 05:53:59,401 INFO [agent-shutdown-hook]
(org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:255) -
Updating checkpoint metadata: logWriteOrderID: 1474685940834, queueSize: 0,
queueHead: 0
24 Sep 2016 05:53:59,407 INFO [agent-shutdown-hook]
(org.apache.flume.channel.file.Log.writeCheckpoint:1034) - Updated checkpoint
for file: ./data/product/log-3 position: 6674 logWriteOrderID: 1474685940834
24 Sep 2016 05:53:59,407 INFO [agent-shutdown-hook]
(org.apache.flume.channel.file.Log.shutdownWorker:857) - Attempting to
shutdown background worker.
24 Sep 2016 05:53:59,409 INFO [agent-shutdown-hook]
(org.apache.flume.channel.file.LogFile$Writer.close:370) - Closing
./data/product/log-3
24 Sep 2016 05:53:59,409 INFO [agent-shutdown-hook]
(org.apache.flume.channel.file.LogFile$RandomReader.close:504) - Closing
RandomReader ./data/product/log-2
24 Sep 2016 05:53:59,414 INFO [agent-shutdown-hook]
(org.apache.flume.channel.file.LogFile$RandomReader.close:504) - Closing
RandomReader ./data/product/log-3
24 Sep 2016 05:53:59,420 INFO [agent-shutdown-hook]
(org.apache.flume.instrumentation.MonitoredCounterGroup.stop:150) - Component
type: CHANNEL, name: channel_etl_product stopped
please let me know if need more information.
thanks
was (Author: hehailin):
hi bellow is the configure file content:
a1.sources = r1
a1.sinks = sink_etl_product sink_kinesis_stream
a1.channels = channel_etl_product channel_kinesis_stream
#source setting
#a1.sources.r1.type = http
a1.sources.r1.type = flume.source.HTTPSource
a1.sources.r1.port = 6558
a1.sources.r1.dataSize = 1
a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = event_type
a1.sources.r1.selector.mapping.etl_product = channel_etl_product
channel_kinesis_stream
#handler setting
a1.sources.r1.handler = flume.source.BlobExtHandler
a1.sources.r1.handler.gzip = true
a1.sources.r1.handler.maxBlobLength = 100000
a1.channels.channel_etl_product.type = file
a1.channels.channel_etl_product.checkpointDir = ./checkpoint/product
a1.channels.channel_etl_product.dataDirs = ./data/product
a1.channels.channel_etl_product.checkpointInterval = 10
#Kinesis Firehose
a1.sinks.sink_etl_mix.type = flume.sink.KinesisFirehoseSink
a1.sinks.sink_etl_mix.channel = channel_etl_product
a1.sinks.sink_etl_mix.streamName = product_flumeFirehoseStream
a1.sinks.sink_etl_mix.endpoint = {firehose_endpoint}
flume.source.HTTPSource,flume.source.BlobExtHandler,flume.sink.KinesisFirehoseSink,
these 3 Java class contains our bussiness logic.
what I want is that the flume will transfer data into AWS Kinesis Firehose.
The application works fine on AWS EC2 linux instance. What I meet is flume
shutdown automaticly after several hours , the process disappear when run "ps
-ef | grep flume".
please let me know if need more information.
thanks
> Flume process shutdown automaticly
> ----------------------------------
>
> Key: FLUME-2996
> URL: https://issues.apache.org/jira/browse/FLUME-2996
> Project: Flume
> Issue Type: Bug
> Environment: Amazon Linux
> Reporter: hailin
>
> hi guys,
> I install flume from apache-flume-1.6.0-bin.tar.gz, and startup flume with
> some my custom configure file with cmd line like this:
> bin/flume-ng agent -Dflume.monitoring.type=http -Dflume.monitoring.port=41414
> -c conf -f conf/flume.conf --name a1 -Dflume.root.logger=INFO,DAILY
> it works fine without any issue, but after several hours when I check the
> process, I found it was shutdown automaticly. Bellow is the flume log
> information:
> .....
> 24 Sep 2016 05:53:59,391 INFO [agent-shutdown-hook]
> (org.apache.flume.lifecycle.LifecycleSupervisor.stop:79) - Stopping
> lifecycle supervisor 10
> 24 Sep 2016 05:53:59,394 INFO [agent-shutdown-hook]
> (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:150) -
> Component type: SINK, name: sink_kinesis_stream stopped
> 24 Sep 2016 05:53:59,394 INFO [agent-shutdown-hook]
> (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:156) - Shutdown
> Metric for type: SINK, name: sink_kinesis_stream. sink.start.time ==
> 1474685921906
> ....
> Can anyone help me fix it? How can I do to stop this issue as our bussiness
> already block because of this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
