Hi Aaron,

Wrt kafka_refresh_time set to 1 sec, please give it a try. I'd be
curious myself about your feedback. In theory it's a valid configuration
and it should work. As extra tests, if 60 secs is not live enough, i may
recommend towards 10, 15 or 20 secs. 

Paolo

On Fri, Feb 17, 2017 at 08:28:11AM -0800, Aaron Finney wrote:
> Hi Paolo,
> 
> That's what I get for becoming our internal package maintainer for pmacct
> by default - updating it now! :D
> 
> One quick sanity check regarding output to Kafka - would setting
> kafka_refresh_time to 1 add significant overhead to nfacctd, e.g. related
> to buffer flushing process/etc?
> 
> My goal is to produce something closer to a live stream, but with
> micro-flows aggregated to 1s granularity, then use Spark to do additional
> transformations/aggregations against local data along with basic anomaly
> detection.
> 
> Thanks!
> 
> Aaron
> 
> 
> On Fri, Feb 17, 2017 at 5:58 AM, Paolo Lucente <pa...@pmacct.net> wrote:
> 
> >
> > Hi Aaron,
> >
> > The feature is post 1.6.1. Can you please switch to master code on GitHub?
> >
> > Thanks,
> > Paolo
> >
> > On Thu, Feb 16, 2017 at 10:44:23AM -0800, Aaron Finney wrote:
> > > Hi Paolo/all,
> > >
> > > I've been unable to get nfacctd to send the Avro schema to a Kafka topic
> > -
> > > I receive the following message when starting nfacctd:
> > >
> > > WARN: [/etc/nfacctd.conf:14] Unknown key: kafka_avro_schema_topic.
> > Ignored.
> > > WARN: [/etc/nfacctd.conf:15] Unknown key: kafka_avro_schema_refresh_
> > time.
> > > Ignored.
> > >
> > > Kafka/Avro are otherwise working; the process successfully writes the
> > > schema to a local file specified in avro_schema_output_file, but I'd like
> > > to have it send to a schema topic in the cluster itself if possible.
> > > Version info is below.
> > >
> > > Thanks!
> > >
> > > Aaron
> > >
> > >
> > > INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.6.1
> > > (20161001-00+c5)
> > > INFO ( default/core ):  '--enable-kafka' '--enable-jansson'
> > '--enable-avro'
> > > 'KAFKA_LIBS=-L/usr/lib64/ -lrdkafka' 'AVRO_CFLAGS=-I/usr/include/avro'
> > > 'AVRO_LIBS=-L/usr/lib -lavro'
> >
> > > _______________________________________________
> > > pmacct-discussion mailing list
> > > http://www.pmacct.net/#mailinglists
> >
> >
> > _______________________________________________
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
> >

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Reply via email to