Updated Branches: refs/heads/flume-1.3.0 f71559067 -> c3bc35619
FLUME-1692: MultiportSyslogTCPSource user documentation and nickname (Mike Percy via Brock Noland) Project: http://git-wip-us.apache.org/repos/asf/flume/repo Commit: http://git-wip-us.apache.org/repos/asf/flume/commit/c3bc3561 Tree: http://git-wip-us.apache.org/repos/asf/flume/tree/c3bc3561 Diff: http://git-wip-us.apache.org/repos/asf/flume/diff/c3bc3561 Branch: refs/heads/flume-1.3.0 Commit: c3bc356196056bebebfd64927f12d04e3b4c0553 Parents: f715590 Author: Brock Noland <[email protected]> Authored: Sat Nov 10 15:46:21 2012 -0600 Committer: Brock Noland <[email protected]> Committed: Sat Nov 10 15:47:52 2012 -0600 ---------------------------------------------------------------------- .../org/apache/flume/conf/source/SourceType.java | 9 ++- flume-ng-doc/sphinx/FlumeUserGuide.rst | 53 +++++++++++++- 2 files changed, 56 insertions(+), 6 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/flume/blob/c3bc3561/flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java ---------------------------------------------------------------------- diff --git a/flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java b/flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java index 426d112..c7c511e 100644 --- a/flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java +++ b/flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java @@ -61,15 +61,20 @@ public enum SourceType { * * @see org.apache.flume.source.SyslogTcpSource */ - SYSLOGTCP("org.apache.flume.source.SyslogTcpSource"), /** + * MultiportSyslogTCPSource + * + * @see org.apache.flume.source.MultiportSyslogTCPSource + */ + MULTIPORT_SYSLOGTCP("org.apache.flume.source.MultiportSyslogTCPSource"), + + /** * SyslogUDPSource * * @see org.apache.flume.source.SyslogUDPSource */ - SYSLOGUDP("org.apache.flume.source.SyslogUDPSource"), /** http://git-wip-us.apache.org/repos/asf/flume/blob/c3bc3561/flume-ng-doc/sphinx/FlumeUserGuide.rst ---------------------------------------------------------------------- diff --git a/flume-ng-doc/sphinx/FlumeUserGuide.rst b/flume-ng-doc/sphinx/FlumeUserGuide.rst index 62cf5f6..ca4ac89 100644 --- a/flume-ng-doc/sphinx/FlumeUserGuide.rst +++ b/flume-ng-doc/sphinx/FlumeUserGuide.rst @@ -818,14 +818,16 @@ Syslog Sources ~~~~~~~~~~~~~~ Reads syslog data and generate Flume events. The UDP source treats an entire -message as a single event. The TCP source on creates a new event for a string -of characters separated by carriage return ('\n'). +message as a single event. The TCP sources create a new event for each string +of characters separated by a newline ('\n'). Required properties are in **bold**. Syslog TCP Source ''''''''''''''''' +The original, tried-and-true syslog TCP source. + ============== =========== ============================================== Property Name Default Description ============== =========== ============================================== @@ -833,14 +835,13 @@ Property Name Default Description **type** -- The component type name, needs to be ``syslogtcp`` **host** -- Host name or IP address to bind to **port** -- Port # to bind to -eventSize 2500 +eventSize 2500 Maximum size of a single event line, in bytes selector.type replicating or multiplexing selector.* replicating Depends on the selector.type value interceptors -- Space separated list of interceptors interceptors.* ============== =========== ============================================== - For example, a syslog TCP source for agent named **agent_foo**: .. code-block:: properties @@ -852,6 +853,49 @@ For example, a syslog TCP source for agent named **agent_foo**: agent_foo.sources.syslogsource-1.host = localhost agent_foo.sources.syslogsource-1.channels = memoryChannel-1 +Multiport Syslog TCP Source +''''''''''''''''''''''''''' + +This is a newer, faster, multi-port capable version of the Syslog TCP source. +Note that the ``ports`` configuration setting has replaced ``port``. +Multi-port capability means that it can listen on many ports at once in an +efficient manner. This source uses the Apache Mina library to do that. +Provides support for RFC-3164 and many common RFC-5424 formatted messages. +Also provides the capability to configure the character set used on a per-port +basis. + +==================== ================ ============================================== +Property Name Default Description +==================== ================ ============================================== +**channels** -- +**type** -- The component type name, needs to be ``multiport_syslogtcp`` +**host** -- Host name or IP address to bind to. +**ports** -- Space-separated list (one or more) of ports to bind to. +eventSize 2500 Maximum size of a single event line, in bytes. +portHeader -- If specified, the port number will be stored in the header of each event using the header name specified here. This allows for interceptors and channel selectors to customize routing logic based on the incoming port. +charset.default UTF-8 Default character set used while parsing syslog events into strings. +charset.port.<port> -- Character set is configurable on a per-port basis. +batchSize 100 Maximum number of events to attempt to process per request loop. Using the default is usually fine. +readBufferSize 1024 Size of the internal Mina read buffer. Provided for performance tuning. Using the default is usually fine. +numProcessors (auto-detected) Number of processors available on the system for use while processing messages. Default is to auto-detect # of CPUs using the Java Runtime API. Mina will spawn 2 request-processing threads per detected CPU, which is often reasonable. +selector.type replicating replicating, multiplexing, or custom +selector.* -- Depends on the ``selector.type`` value +interceptors -- Space separated list of interceptors. +interceptors.* +==================== ================ ============================================== + +For example, a multiport syslog TCP source for agent named **agent_foo**: + +.. code-block:: properties + + agent_foo.sources = src-0 + agent_foo.channels = ch-0 + agent_foo.sources.src-0.type = multiport_syslogtcp + agent_foo.sources.src-0.channels = ch-0 + agent_foo.sources.src-0.host = 0.0.0.0 + agent_foo.sources.src-0.ports = 10001 10002 10003 + agent_foo.sources.src-0.portHeader = port + Syslog UDP Source ''''''''''''''''' @@ -2428,6 +2472,7 @@ org.apache.flume.Source NETCAT org.apache.flume.s org.apache.flume.Source SEQ org.apache.flume.source.SequenceGeneratorSource org.apache.flume.Source EXEC org.apache.flume.source.ExecSource org.apache.flume.Source SYSLOGTCP org.apache.flume.source.SyslogTcpSource +org.apache.flume.Source MULTIPORT_SYSLOGTCP org.apache.flume.source.MultiportSyslogTCPSource org.apache.flume.Source SYSLOGUDP org.apache.flume.source.SyslogUDPSource org.apache.flume.Source -- org.apache.flume.source.avroLegacy.AvroLegacySource org.apache.flume.Source -- org.apache.flume.source.thriftLegacy.ThriftLegacySource
