damccorm opened a new issue, #21047:
URL: https://github.com/apache/beam/issues/21047
Reason:
We have dataflow to call one internal API (the API call take 60ms). So we
want to start more threads to call this API. Kafka IO thread is control by
split by default. And split is control by total partitions and system
suggestion number. And system suggestion number is 4 * max workers. which is
too low for us.
Solution:
Use facture to allow user inheritance KafkaUnboundedSource. and override
the split function like
```
// code placeholder
@Override
public List<KafkaUnboundedSource<K, V>> split(int desiredNumSplits,
PipelineOptions options)
throws Exception {
int split = this.getMaxSplits();
if(split <= 0){
return super.split(desiredNumSplits, options);
}else{
return super.split(split, options);
}
}
```
this may also useful for Override other feature of the KafkaUnboundedSource
without copy all the io code.
Imported from Jira
[BEAM-12631](https://issues.apache.org/jira/browse/BEAM-12631). Original Jira
may contain additional context.
Reported by: renhongxiang.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
