[
https://issues.apache.org/jira/browse/KAFKA-257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194389#comment-13194389
]
Sam Shah commented on KAFKA-257:
--------------------------------
Thanks Neha. Answers:
1. Done. See updated patch.
2. The KafkaRecordWriter queues up messages up to a configurable amount
(default 10MB) so it is amortizing network bandwidth. It doesn't make sense to
start up an async background thread, as the task will have to block anyway to
push data (it's also not a good idea from a node utilization perspective.)
1. Yup, that's a good point, the old hadoop producer only supported one broker
in its URI and I didn't fix it. The updated patch allows multiple brokers
separated by commas.
2. Async shouldn't be used (see point #2 above), which is why I didn't add
support for the other options.
3. I haven't come across a use case yet for a custom partitioner. I can add
support later; it should be easy.
> Hadoop producer should use software load balancer
> -------------------------------------------------
>
> Key: KAFKA-257
> URL: https://issues.apache.org/jira/browse/KAFKA-257
> Project: Kafka
> Issue Type: Improvement
> Components: contrib
> Affects Versions: 0.7
> Reporter: Sam Shah
> Fix For: 0.7.1
>
> Attachments: kafka-257.patch
>
>
> Improvements to Kafka's Hadoop producer:
> * Uses new Producer API (as opposed to SyncProducer)
> * Supports software load balancer (Kafka URI for this is
> kafka+zk://<zk-path>#<topic-name>)
> * Can set compression codec (default is 0 or no compression)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
