Alex, This is the overview on what I have done.
Windows (running Flume agent inside JVM. in cmd.exe: java log4J.configuration=.... -cp "......\lib\*" org.apache.flume.node.Application -f ... -n... ) source: exec running tail command sink: avro bind to a localhost/port putty configured to pipe from local host/port to remote host/port EC2 AMI (running Flume agent script:$ flume-ng agent -f.. -n .....) Source: avro, listening on the port mentioned above Sink: hdfs Every time I would append something to a log file on Windows box, it will get pushed and stored in hdfs. Rgds, -----Original Message----- From: Alexander Alten-Lorenz [mailto:[email protected]] Sent: Thursday, December 20, 2012 2:03 AM To: [email protected] Subject: Re: Flume to stream logs live Hey AK47, good to know, can you describe for the community what exactly you've done and what you've archived? Would be great, cheers, Alex On Dec 19, 2012, at 5:20 PM, "Kartashov, Andy" <[email protected]> wrote: > Thank you for your suggestions: Mr. Chavez , Mr. Lorenz, Mr. Dechoux and Mr. > Connolly, > > I just managed to successfully configure flume to push logs from windows box, > running behind Proxy, to the Avro-agent, running on Linux box, and without > Cygwin.. by just running JVM process. So happy. > > Cheers, > AK47 > > > -----Original Message----- > From: Alexander Lorenz [mailto:[email protected]] > Sent: Monday, December 17, 2012 2:58 AM > To: [email protected] > Subject: Re: Flume to stream logs live > > Hey, > > fluentd is a good tip, they have a custom sink: > https://cwiki.apache.org/confluence/display/FLUME/Flume+NG+Plugins > > This could work. > > Best, > Alex > > On Dec 17, 2012, at 2:33 AM, Juhani Connolly > <[email protected]> wrote: > >> If you can't get flume running on windows your alternatives look like >> syslog, scribed and fluentd. We used scribed before moving to flume, but >> tbh, it can be a pain to compile and the hdfs support on it is not the best, >> we had data loss if there was an hdfs restart etc. I can't say much about >> the other two >> >> On 12/14/2012 11:47 PM, Kartashov, Andy wrote: >>> >>> Flummers, >>> >>> Loved working with Flume 1.2 -- very easy and simple configuration, it was >>> a pleasure to work with. Managed to "tail --F" logs from unix server and >>> into a hdfs cluster. The problem started when I also needed to push logs >>> from a Windows application server. Spent three days researching on how to >>> install flume on Windows and run a deamon/agent that will push the logs to >>> the Avro source I successfully configured and ran on Unix. No luck. So I am >>> looking t alternative. Is there other framework available out there to help >>> me with my issue. What about scribe? >>> > >>> > >>> >>> NOTICE: This e-mail message and any attachments are confidential, subject >>> to copyright and may be privileged. Any unauthorized use, copying or >>> disclosure is prohibited. If you are not the intended recipient, please >>> delete and contact the sender immediately. Please consider the environment >>> before printing this e-mail. AVIS : le présent courriel et toute pièce >>> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur >>> et peuvent être couverts par le secret professionnel. Toute utilisation, >>> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le >>> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement >>> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent >>> courriel >> > > NOTICE: This e-mail message and any attachments are confidential, subject to > copyright and may be privileged. Any unauthorized use, copying or disclosure > is prohibited. If you are not the intended recipient, please delete and > contact the sender immediately. Please consider the environment before > printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui > l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent > être couverts par le secret professionnel. Toute utilisation, copie ou > divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire > prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. > Veuillez penser à l'environnement avant d'imprimer le présent courriel NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
