[
https://issues.apache.org/jira/browse/FLUME-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463428#comment-13463428
]
Denny Ye commented on FLUME-1574:
---------------------------------
Flume is as blank box for client. It may hard to explain the 'data loss' to
client from unreachable missing selection covered by detailed Flume structure.
Attached is my flume.conf file with custom ChannelSelector and Interceptor
{code:xml}
//Generated by code
agent.sources = e2e-src scribesource
agent.channels = e2e-ch hdfs-fc-1 hdfs-fc-2 hdfs-fc-3 hdfs-fc-4 hdfs-fc-5
agent.sinks = e2e-dst hdfsSink-1 hdfsSink-2 hdfsSink-3 hdfsSink-4 hdfsSink-5
#E2E Tracking
agent.sources.e2e-src.type = com.aaa.XXXXX.flume.E2ETrackSource
agent.sources.e2e-src.sampleInterval = 60
agent.sources.e2e-src.channels = e2e-ch
agent.sinks.e2e-dst.type = org.apache.flume.sink.hdfs.HDFSEventSink
agent.sinks.e2e-dst.hdfs.fileType = DataStream
agent.sinks.e2e-dst.hdfs.txnEventMax = 1
agent.sinks.e2e-dst.hdfs.batchSize = 10
agent.sinks.e2e-dst.hdfs.rollInterval = 3600
agent.sinks.e2e-dst.hdfs.rollCount = 1000000
agent.sinks.e2e-dst.hdfs.rollSize = 0
agent.sinks.e2e-dst.hdfs.path = /XXXXX/_pendingS_/e2etrack/dt=%{dateStr}
agent.sinks.e2e-dst.channel = e2e-ch
#agent.channels.e2e-ch.type = org.apache.flume.channel.PseudoTxnMemoryChannel
agent.channels.e2e-ch.type = com.aaa.XXXXX.flume.E2EMemoryChannel
#Sources
agent.sources.scribesource.type = org.apache.flume.source.scribe.ScribeSource
agent.sources.scribesource.interceptors= host stamp datasource
agent.sources.scribesource.interceptors.host.type = host
agent.sources.scribesource.interceptors.stamp.type = timestamp
agent.sources.scribesource.interceptors.datasource.type =
com.aaa.XXXXX.flume.DataSourceInterceptor$Builder
agent.sources.scribesource.interceptors.datasource.apiAddr = /agent.properties
agent.sources.scribesource.selector.type =
com.aaa.XXXXX.flume.DataSourceSelector
agent.sources.scribesource.channels = hdfs-fc-1 hdfs-fc-2 hdfs-fc-3 hdfs-fc-4
hdfs-fc-5
#Sinks
agent.sinks.hdfsSink-1.type = org.apache.flume.sink.hdfs.HDFSEventSink
agent.sinks.hdfsSink-1.hdfs.fileType = DataStream
agent.sinks.hdfsSink-1.hdfs.txnEventMax = 1000
agent.sinks.hdfsSink-1.hdfs.batchSize = 1000000
agent.sinks.hdfsSink-1.hdfs.rollInterval = 3600
agent.sinks.hdfsSink-1.hdfs.rollCount = 1000000
agent.sinks.hdfsSink-1.hdfs.rollSize = 0
agent.sinks.hdfsSink-1.hdfs.path = /XXXXX/_pending_/%{yId}/dt=%{dateStr}
agent.sinks.hdfsSink-1.hdfs.filePrefix = %H-%{host}
agent.sinks.hdfsSink-1.channel = hdfs-fc-1
agent.sinks.hdfsSink-2.type = org.apache.flume.sink.hdfs.HDFSEventSink
agent.sinks.hdfsSink-2.hdfs.fileType = DataStream
agent.sinks.hdfsSink-2.hdfs.txnEventMax = 1000
agent.sinks.hdfsSink-2.hdfs.batchSize = 1000000
agent.sinks.hdfsSink-2.hdfs.rollInterval = 3600
agent.sinks.hdfsSink-2.hdfs.rollCount = 1000000
agent.sinks.hdfsSink-2.hdfs.rollSize = 0
agent.sinks.hdfsSink-2.hdfs.path = /XXXXX/_pending_/%{yId}/dt=%{dateStr}
agent.sinks.hdfsSink-2.hdfs.filePrefix = %H-%{host}
agent.sinks.hdfsSink-2.channel = hdfs-fc-2
agent.sinks.hdfsSink-3.type = org.apache.flume.sink.hdfs.HDFSEventSink
agent.sinks.hdfsSink-3.hdfs.fileType = DataStream
agent.sinks.hdfsSink-3.hdfs.txnEventMax = 1000
agent.sinks.hdfsSink-3.hdfs.batchSize = 1000000
agent.sinks.hdfsSink-3.hdfs.rollInterval = 3600
agent.sinks.hdfsSink-3.hdfs.rollCount = 1000000
agent.sinks.hdfsSink-3.hdfs.rollSize = 0
agent.sinks.hdfsSink-3.hdfs.path = /XXXXX/_pending_/%{yId}/dt=%{dateStr}
agent.sinks.hdfsSink-3.hdfs.filePrefix = %H-%{host}
agent.sinks.hdfsSink-3.channel = hdfs-fc-3
agent.sinks.hdfsSink-4.type = org.apache.flume.sink.hdfs.HDFSEventSink
agent.sinks.hdfsSink-4.hdfs.fileType = DataStream
agent.sinks.hdfsSink-4.hdfs.txnEventMax = 1000
agent.sinks.hdfsSink-4.hdfs.batchSize = 1000000
agent.sinks.hdfsSink-4.hdfs.rollInterval = 3600
agent.sinks.hdfsSink-4.hdfs.rollCount = 1000000
agent.sinks.hdfsSink-4.hdfs.rollSize = 0
agent.sinks.hdfsSink-4.hdfs.path = /XXXXX/_pending_/%{yId}/dt=%{dateStr}
agent.sinks.hdfsSink-4.hdfs.filePrefix = %H-%{host}
agent.sinks.hdfsSink-4.channel = hdfs-fc-4
agent.sinks.hdfsSink-5.type = org.apache.flume.sink.hdfs.HDFSEventSink
agent.sinks.hdfsSink-5.hdfs.fileType = DataStream
agent.sinks.hdfsSink-5.hdfs.txnEventMax = 1000
agent.sinks.hdfsSink-5.hdfs.batchSize = 1000000
agent.sinks.hdfsSink-5.hdfs.rollInterval = 3600
agent.sinks.hdfsSink-5.hdfs.rollCount = 1000000
agent.sinks.hdfsSink-5.hdfs.rollSize = 0
agent.sinks.hdfsSink-5.hdfs.path = /XXXXX/_pendingS_/%{yId}/dt=%{dateStr}
agent.sinks.hdfsSink-5.hdfs.filePrefix = %H-%{host}
agent.sinks.hdfsSink-5.channel = hdfs-fc-5
#Channels
agent.channels.hdfs-fc-1.capacity = 1000000
agent.channels.hdfs-fc-1.transactionCapacity = 1000
agent.channels.hdfs-fc-1.type = org.apache.flume.channel.file.FileChannel
agent.channels.hdfs-fc-1.dataDirs = /var/flume/fc1/data
agent.channels.hdfs-fc-1.checkpointDir = /var/flume/fc1/checkpoint
agent.channels.hdfs-fc-2.capacity = 1000000
agent.channels.hdfs-fc-2.transactionCapacity = 1000
agent.channels.hdfs-fc-2.type = org.apache.flume.channel.file.FileChannel
agent.channels.hdfs-fc-2.dataDirs = /var/flume/fc2/data
agent.channels.hdfs-fc-2.checkpointDir = /var/flume/fc2/checkpoint
agent.channels.hdfs-fc-3.capacity = 1000000
agent.channels.hdfs-fc-3.transactionCapacity = 1000
agent.channels.hdfs-fc-3.type = org.apache.flume.channel.file.FileChannel
agent.channels.hdfs-fc-3.dataDirs = /var/flume/fc3/data
agent.channels.hdfs-fc-3.checkpointDir = /var/flume/fc3/checkpoint
agent.channels.hdfs-fc-4.capacity = 1000000
agent.channels.hdfs-fc-4.transactionCapacity = 1000
agent.channels.hdfs-fc-4.type = org.apache.flume.channel.file.FileChannel
agent.channels.hdfs-fc-4.dataDirs = /var/flume/fc4/data
agent.channels.hdfs-fc-4.checkpointDir = /var/flume/fc4/checkpoint
agent.channels.hdfs-fc-5.capacity = 1000000
agent.channels.hdfs-fc-5.transactionCapacity = 1000
agent.channels.hdfs-fc-5.type = org.apache.flume.channel.file.FileChannel
agent.channels.hdfs-fc-5.dataDirs = /var/flume/fc5/data
agent.channels.hdfs-fc-5.checkpointDir = /var/flume/fc5/checkpoint
{code}
> Records the probable data loss while missing selected channel
> -------------------------------------------------------------
>
> Key: FLUME-1574
> URL: https://issues.apache.org/jira/browse/FLUME-1574
> Project: Flume
> Issue Type: Improvement
> Components: Channel
> Affects Versions: v1.2.0
> Reporter: Denny Ye
> Assignee: Denny Ye
> Priority: Minor
>
> When I'm using custom ChannelSelector, there have many chances that it cannot
> select any channels to put events into. Both required and optional channels
> cannot be chosen.
> Event has been discarded and client received successful message.
> It's the data loss case from missing selection.
> In my opinion, recording this case into log might be better method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
