Re: Kafka 0.11.0.1 and filebeat 6.1 compatibility

[Ewen Cheslack-Postava]

filebeat is implemented using sarama last I checked, so presumably they are
on a version that doesn't know about Kafka 0.11.0.1 and therefore it
doesn't know which API versions to use.

Not sure if they support leaving it blank or exactly how the sarama config
works, but as far as I know sarama has support for the API compatibility
request that was added way back in 0.10.0.0, meaning it really shouldn't
need version info specified for anything newer than that unless they
specifically chose not to support some API versions.

In either case, this is probably better answered by folks from Elastic or
the sarama project.

-Ewen

On Fri, Feb 9, 2018 at 7:11 AM, Sandeep Sarkar 
wrote:

> Hi All,
>
>
>
> I am using filebeat 6.1 and kafka 0.11.0.1 to push logs. From filebeat
> logs I could see that communication is getting established but then logs
> are not getting pushed.
>
> Also when I updated filebeat's output.kafka.version to 0.11.0.1 I get
> error message :
>
>
>
> 2018/02/08 09:41:15.417691 beat.go:635: CRIT Exiting: error initializing
> publisher:  unknown/unsupported kafka version '0.11.0.1' accessing
> 'output.kafka' (source:'/etc/filebeat/filebeat.yml')
>
> Exiting: error initializing publisher: unknown/unsupported kafka version
> '0.11.0.1' accessing 'output.kafka' (source:'/etc/filebeat/filebeat.yml')
>
>
>
> So I used kafka 0.10.2.0 and updated filebeat's output.kafka.version to
> 0.10.2.0 and the error message is gone but I do not get any logs in kafka.
> Filebeate throws
>
> Kafka publish failed with: circuit breaker is open
>
>
>
> And after some time it throws below message in a loop:
>
> 2018/02/09 15:08:13.075932 client.go:234: DBG [kafka] Kafka publish failed
> with: circuit breaker is open
>
> 2018/02/09 15:08:13.110793 client.go:220: DBG [kafka] finished kafka batch
>
> 2018/02/09 15:08:13.110814 client.go:234: DBG [kafka] Kafka publish failed
> with: circuit breaker is open
>
> 2018/02/09 15:08:13.149482 metrics.go:39: INFO Non-zero metrics in the
> last 30s: beat.info.uptime.ms=3 beat.memstats.gc_next=18583840
> beat.memstats.memory_alloc=17667464 beat.memstats.memory_total=3883962925856
> filebeat.harvester.open_files=11 filebeat.harvester.running=10
> libbeat.config.module.running=0 libbeat.output.events.batches=832
> libbeat.output.events.failed=1703936 libbeat.output.events.total=1703936
> libbeat.pipeline.clients=2 libbeat.pipeline.events.active=4118
> libbeat.pipeline.events.retry=1703936 registrar.states.current=21
>
>
>
>
>
> Please tell me how to debug this.
>
>
>
> I am using logstash 6.1 as consumer.
>
>
>
> --
> Thanks
>
> http://www.oracle.com/
> Sandeep Sarkar | Member of Technical Staff
> Phone: HYPERLINK "tel:+918067445685"+918067445685
> Oracle CGBU PM
>
> Oracle India Bangalore
>
> http://www.oracle.com/commitment
>
> Oracle is committed to developing practices and products that help protect
> the environment
>
>
>
>
>

Kafka 0.11.0.1 and filebeat 6.1 compatibility

[Sandeep Sarkar]

Hi All,

 

I am using filebeat 6.1 and kafka 0.11.0.1 to push logs. From filebeat logs I 
could see that communication is getting established but then logs are not 
getting pushed.

Also when I updated filebeat's output.kafka.version to 0.11.0.1 I get error 
message :

 

2018/02/08 09:41:15.417691 beat.go:635: CRIT Exiting: error initializing 
publisher:  unknown/unsupported kafka version '0.11.0.1' accessing 
'output.kafka' (source:'/etc/filebeat/filebeat.yml')

Exiting: error initializing publisher: unknown/unsupported kafka version 
'0.11.0.1' accessing 'output.kafka' (source:'/etc/filebeat/filebeat.yml')

 

So I used kafka 0.10.2.0 and updated filebeat's output.kafka.version to 
0.10.2.0 and the error message is gone but I do not get any logs in kafka. 
Filebeate throws 

Kafka publish failed with: circuit breaker is open

 

And after some time it throws below message in a loop:

2018/02/09 15:08:13.075932 client.go:234: DBG [kafka] Kafka publish failed 
with: circuit breaker is open

2018/02/09 15:08:13.110793 client.go:220: DBG [kafka] finished kafka batch

2018/02/09 15:08:13.110814 client.go:234: DBG [kafka] Kafka publish failed 
with: circuit breaker is open

2018/02/09 15:08:13.149482 metrics.go:39: INFO Non-zero metrics in the last 
30s: beat.info.uptime.ms=3 beat.memstats.gc_next=18583840 
beat.memstats.memory_alloc=17667464 beat.memstats.memory_total=3883962925856 
filebeat.harvester.open_files=11 filebeat.harvester.running=10 
libbeat.config.module.running=0 libbeat.output.events.batches=832 
libbeat.output.events.failed=1703936 libbeat.output.events.total=1703936 
libbeat.pipeline.clients=2 libbeat.pipeline.events.active=4118 
libbeat.pipeline.events.retry=1703936 registrar.states.current=21

 

 

Please tell me how to debug this.

 

I am using logstash 6.1 as consumer.

 

-- 
Thanks

http://www.oracle.com/
Sandeep Sarkar | Member of Technical Staff
Phone: HYPERLINK "tel:+918067445685"+918067445685 
Oracle CGBU PM

Oracle India Bangalore 

http://www.oracle.com/commitment

Oracle is committed to developing practices and products that help protect the 
environment