[
https://issues.apache.org/jira/browse/FLUME-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitaly updated FLUME-2430:
--------------------------
Description:
Not all events are delivered and persisted to file when using multi agent flow:
# agent1
agent1.sources = avroSource1
agent1.sinks = avroSync
agent1.channels = memoryChannel
# Describe/configure the source
agent1.sources.avroSource1.type = avro
agent1.sources.avroSource1.bind = localhost
agent1.sources.avroSource1.port = 41414
# Describe the sink
agent1.sinks.avroSync.type = avro
agent1.sinks.avroSync.hostname = localhost
agent1.sinks.avroSync.port = 4444
# Use a channel which buffers events in memory
agent1.channels.memoryChannel.type = memory
agent1.channels.memoryChannel.capacity = 10000
agent1.channels.memoryChannel.transactionCapacity = 1000
# Bind the source and sink to the channel
agent1.sources.avroSource1.channels = memoryChannel
agent1.sinks.avroSync.channel = memoryChannel
# agent2
agent2.sources = avroSource
agent2.sinks = fileRollSink
agent2.channels = memoryChannel
# Describe/configure the source
agent2.sources.avroSource.type = avro
agent2.sources.avroSource.bind = localhost
agent2.sources.avroSource.port = 4444
# Describe the sink
agent2.sinks.fileRollSink.type = file_roll
agent2.sinks.fileRollSink.sink.directory = C:\\Users\\flume\\flumeFileAgent
agent2.sinks.fileRollSink.sink.rollInterval=0
# Use a channel which buffers events in memory
agent2.channels.memoryChannel.type = memory
agent2.channels.memoryChannel.capacity = 10000
agent2.channels.memoryChannel.transactionCapacity = 1000
# Bind the source and sink to the channel
agent2.sources.avroSource.channels = memoryChannel
agent2.sinks.fileRollSink.channel = memoryChannel
For example I am passing 1000 events to the first "agent1" but "agent2" doesn't
receive from "agent1" all of them, only 800 or 700 are stored in file. No
exceptions are thrown by flume agents.
was:
Not all events are delivered and persisted to file when using multi agent flow:
# agent1
agent1.sources = avroSource1
agent1.sinks = avroSync
agent1.channels = memoryChannel
# Describe/configure the source
agent1.sources.avroSource1.type = avro
agent1.sources.avroSource1.bind = localhost
agent1.sources.avroSource1.port = 41414
# Describe the sink
agent1.sinks.avroSync.type = avro
agent1.sinks.avroSync.hostname = localhost
agent1.sinks.avroSync.port = 4444
# Use a channel which buffers events in memory
agent1.channels.memoryChannel.type = memory
agent1.channels.memoryChannel.capacity = 10000
agent1.channels.memoryChannel.transactionCapacity = 1000
# Bind the source and sink to the channel
agent1.sources.avroSource1.channels = memoryChannel
agent1.sinks.avroSync.channel = memoryChannel
# agent2
agent2.sources = avroSource
agent2.sinks = fileRollSink
agent2.channels = memoryChannel
# Describe/configure the source
agent2.sources.avroSource.type = avro
agent2.sources.avroSource.bind = localhost
agent2.sources.avroSource.port = 4444
# Describe the sink
agent2.sinks.fileRollSink.type = file_roll
agent2.sinks.fileRollSink.sink.directory = C:\\Users\\flume\\flumeFileAgent
agent2.sinks.fileRollSink.sink.rollInterval=0
# Use a channel which buffers events in memory
agent2.channels.memoryChannel.type = memory
agent2.channels.memoryChannel.capacity = 10000
agent2.channels.memoryChannel.transactionCapacity = 1000
# Bind the source and sink to the channel
agent2.sources.avroSource.channels = memoryChannel
agent2.sinks.fileRollSink.channel = memoryChannel
For example I am passing 1000 events to the first "agent1" but "agent2" doesn't
receive from "agent1" all of them, only 800 or 700 are stored in file. No
exceptions are thrown by flume agents.
> Losing events with multiple agents flow
> ---------------------------------------
>
> Key: FLUME-2430
> URL: https://issues.apache.org/jira/browse/FLUME-2430
> Project: Flume
> Issue Type: Bug
> Affects Versions: v1.5.0, v1.5.0.1
> Environment: Win7 64bit, Cygwin
> Reporter: Vitaly
>
> Not all events are delivered and persisted to file when using multi agent
> flow:
>
> # agent1
> agent1.sources = avroSource1
> agent1.sinks = avroSync
> agent1.channels = memoryChannel
> # Describe/configure the source
> agent1.sources.avroSource1.type = avro
> agent1.sources.avroSource1.bind = localhost
> agent1.sources.avroSource1.port = 41414
> # Describe the sink
> agent1.sinks.avroSync.type = avro
> agent1.sinks.avroSync.hostname = localhost
> agent1.sinks.avroSync.port = 4444
> # Use a channel which buffers events in memory
> agent1.channels.memoryChannel.type = memory
> agent1.channels.memoryChannel.capacity = 10000
> agent1.channels.memoryChannel.transactionCapacity = 1000
> # Bind the source and sink to the channel
> agent1.sources.avroSource1.channels = memoryChannel
> agent1.sinks.avroSync.channel = memoryChannel
> # agent2
> agent2.sources = avroSource
> agent2.sinks = fileRollSink
> agent2.channels = memoryChannel
> # Describe/configure the source
> agent2.sources.avroSource.type = avro
> agent2.sources.avroSource.bind = localhost
> agent2.sources.avroSource.port = 4444
> # Describe the sink
> agent2.sinks.fileRollSink.type = file_roll
> agent2.sinks.fileRollSink.sink.directory = C:\\Users\\flume\\flumeFileAgent
> agent2.sinks.fileRollSink.sink.rollInterval=0
> # Use a channel which buffers events in memory
> agent2.channels.memoryChannel.type = memory
> agent2.channels.memoryChannel.capacity = 10000
> agent2.channels.memoryChannel.transactionCapacity = 1000
> # Bind the source and sink to the channel
> agent2.sources.avroSource.channels = memoryChannel
> agent2.sinks.fileRollSink.channel = memoryChannel
> For example I am passing 1000 events to the first "agent1" but "agent2"
> doesn't receive from "agent1" all of them, only 800 or 700 are stored in
> file. No exceptions are thrown by flume agents.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
