Xu Mingmin created BEAM-6285:
--------------------------------
Summary: option to disable offsetConsumer in KafkaIO
Key: BEAM-6285
URL: https://issues.apache.org/jira/browse/BEAM-6285
Project: Beam
Issue Type: Improvement
Components: io-java-kafka
Reporter: Xu Mingmin
Assignee: Xu Mingmin
Add an option in KafkaIO.read() to avoid using `offsetConsumer`.
*What's the problem:*
When security is enabled, function `KafkaUnboundedReader.updateLatestOffsets()`
always fails with lots of WARN log.
*What's the cause:*
`offsetConsumer` uses a mocked `ConsumerConfig.GROUP_ID_CONFIG`, and it doesn't
work as expected when security is enabled. In our case,
`ConsumerConfig.GROUP_ID_CONFIG` is the key to validate whether this consumer
is approved.
*What's the solution:*
Add an option to disable `offsetConsumer` by developers;
*What's the side effect:*
With `offsetConsumer` disabled, backlog is unknown.
[~rangadi] any comments?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
