filebeat is implemented using sarama last I checked, so presumably they are
on a version that doesn't know about Kafka 0.11.0.1 and therefore it
doesn't know which API versions to use.

Not sure if they support leaving it blank or exactly how the sarama config
works, but as far as I know sarama has support for the API compatibility
request that was added way back in 0.10.0.0, meaning it really shouldn't
need version info specified for anything newer than that unless they
specifically chose not to support some API versions.

In either case, this is probably better answered by folks from Elastic or
the sarama project.

-Ewen

On Fri, Feb 9, 2018 at 7:11 AM, Sandeep Sarkar <sandeep.sar...@oracle.com>
wrote:

> Hi All,
>
>
>
> I am using filebeat 6.1 and kafka 0.11.0.1 to push logs. From filebeat
> logs I could see that communication is getting established but then logs
> are not getting pushed.
>
> Also when I updated filebeat's output.kafka.version to 0.11.0.1 I get
> error message :
>
>
>
> 2018/02/08 09:41:15.417691 beat.go:635: CRIT Exiting: error initializing
> publisher:  unknown/unsupported kafka version '0.11.0.1' accessing
> 'output.kafka' (source:'/etc/filebeat/filebeat.yml')
>
> Exiting: error initializing publisher: unknown/unsupported kafka version
> '0.11.0.1' accessing 'output.kafka' (source:'/etc/filebeat/filebeat.yml')
>
>
>
> So I used kafka 0.10.2.0 and updated filebeat's output.kafka.version to
> 0.10.2.0 and the error message is gone but I do not get any logs in kafka.
> Filebeate throws
>
> Kafka publish failed with: circuit breaker is open
>
>
>
> And after some time it throws below message in a loop:
>
> 2018/02/09 15:08:13.075932 client.go:234: DBG [kafka] Kafka publish failed
> with: circuit breaker is open
>
> 2018/02/09 15:08:13.110793 client.go:220: DBG [kafka] finished kafka batch
>
> 2018/02/09 15:08:13.110814 client.go:234: DBG [kafka] Kafka publish failed
> with: circuit breaker is open
>
> 2018/02/09 15:08:13.149482 metrics.go:39: INFO Non-zero metrics in the
> last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=18583840
> beat.memstats.memory_alloc=17667464 beat.memstats.memory_total=3883962925856
> filebeat.harvester.open_files=11 filebeat.harvester.running=10
> libbeat.config.module.running=0 libbeat.output.events.batches=832
> libbeat.output.events.failed=1703936 libbeat.output.events.total=1703936
> libbeat.pipeline.clients=2 libbeat.pipeline.events.active=4118
> libbeat.pipeline.events.retry=1703936 registrar.states.current=21
>
>
>
>
>
> Please tell me how to debug this.
>
>
>
> I am using logstash 6.1 as consumer.
>
>
>
> --
> Thanks
>
> http://www.oracle.com/
> Sandeep Sarkar | Member of Technical Staff
> Phone: HYPERLINK "tel:+918067445685"+918067445685
> Oracle CGBU PM&C
>
> Oracle India Bangalore
>
> http://www.oracle.com/commitment
>
> Oracle is committed to developing practices and products that help protect
> the environment
>
>
>
>
>

Reply via email to