I will check with the QA engineer who ran our tests. However, you might want to test with Flume 1.3.0 which was just released. We actually use the CDH version of Flume from Cloudera and so our tests had many of the fixes that are in 1.3. I'm not sure what if what we were measuring is what you are. We tested the performance difference is what the application sees, not the time to make it to the repository.
In our tests the remote agent is running on a remote machine where the latency is most likely the primary reason the embedded appender performs so much better. Also, since you are using a vanilla pattern layout your log events may be smaller than ours. We are using StructuredDataMessages and each of the items in the Message are added as a header to the FlumeEvent, in addition to being included in the message. This allows them to be efficiently inserted into Cassandra. Ralph On Dec 4, 2012, at 3:44 AM, SimonMo wrote: > Hi Ralph, > > can u give me your properties for this test. > i've got the same speed for both appenders. > and the second problem with the embedded: if i send 20.000 logs i only get > ~17.000 logs. > > embedded: > <Flume name="embedded" suppressExceptions="false" > compress="false" > embedded="true" batchsize="1"> > <Property name="channels">file</Property> > <Property name="channels.file.type">file</Property> > <Property > name="channels.file.checkpointDir">filechannel/checkpoint</Property> > <Property > name="channels.file.dataDirs">filechannel/data</Property> > <Property name="sinks">agent1</Property> > <Property name="sinks.agent1.channel">file</Property> > <Property name="sinks.agent1.type">avro</Property> > <Property > name="sinks.agent1.hostname">127.0.0.1</Property> > <Property name="sinks.agent1.port">41414</Property> > <PatternLayout pattern="%m" /> > </Flume> > > not embedded: > <Flume name="flumeAppender" suppressExceptions="false" > compress="false" > embedded="false"> > <Agent host="127.0.0.1" port="42424" /> > <PatternLayout> > <pattern>%m</pattern> > </PatternLayout> > </Flume> > > Regards, > Simon > > > > > > -- > View this message in context: > http://apache-logging.6191.n7.nabble.com/Flume-Appender-tp7089p35073.html > Sent from the Log4j - Users mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org > For additional commands, e-mail: log4j-user-h...@logging.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org For additional commands, e-mail: log4j-user-h...@logging.apache.org
