Answers inline: >On Thu, Oct 25, 2012 at 11:54 AM, Paul Chavez ><[email protected]> wrote: >> I have tried to go the syslogUDP route to get log files from a Windows >> server to a flume agent, and did not find it an adequate solution. >> >> - We are seeing corruped events when sending IIS logs (known issue: >> https://issues.apache.org/jira/browse/FLUME-1365) >> - Our data is too large to fit in a 1500 byte ethernet frame so events are >> fragmented and the syslogUDP source ignores the continuation packets, >> resulting in truncated events. > >I am not a Syslog expert, is there a way to detect the continuation packets? >If so I think we should open a JIRA on this issue.
I am no expert either but the various syslog related RFC and RFC-type documentation I can find recommends that messeages be kept small in order to avoid fragmentation. >From an IP perspective, there are Ipv4 header flags that indicate if more >fragments follow along with a fragment identifier to help reassemble >out-of-order packets so it certainly is possible even if it goes against RFC >recommendations. Testing with the syslogTCP source did not show any issues with fragmentation, but the tool we are using to send syslog messages over TCP (LogParser) does not separate messages with a carriage return so messages weren't parsed correctly by the flume source. >> I have just been able to build the flume-ng agent for Windows and am testing >> the avro-client functionality on Windows. I think this will be the best bet >> for us in the short term, using LogParser to incrementally create files via >> scheduled task for the avro client to send along. >> >> Long term we want to develop a .net Avro client library for our apps to use >> directly. I suppose a log4net avro appender would be nice too. > >I think a .NET version of the RPCClient and log4net appender would be awesome. > If you do write them, would you considering pushing them back to the flume >community? I would hope so but I am not in a position to make any guarantees on behalf of my employer.
