[jira] [Commented] (KAFKA-1935) Consumer should use a separate socket for Coordinator connection
[ https://issues.apache.org/jira/browse/KAFKA-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878366#comment-15878366 ] Dipen Patel commented on KAFKA-1935: Thanks for the clarifications. If this isn't a good issue for me to be working on at the moment, I can definitely find another issue to get myself acquainted with the codebase. > Consumer should use a separate socket for Coordinator connection > > > Key: KAFKA-1935 > URL: https://issues.apache.org/jira/browse/KAFKA-1935 > Project: Kafka > Issue Type: Sub-task >Reporter: Guozhang Wang > Labels: newbie > > KAFKA-1925 is just a quick-fix of this issue, we need to let consumer to be > able to create separate sockets for the same server for coordinator / broker > roles. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-1935) Consumer should use a separate socket for Coordinator connection
[ https://issues.apache.org/jira/browse/KAFKA-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876596#comment-15876596 ] Jason Gustafson commented on KAFKA-1935: NetworkClient already allows multiple in-flight requests (the producer uses this heavily). bq. I'm not sure if this is a good newbie JIRA or not... what do you think? Yeah, probably not, and at the moment, the benefit seems fairly small. > Consumer should use a separate socket for Coordinator connection > > > Key: KAFKA-1935 > URL: https://issues.apache.org/jira/browse/KAFKA-1935 > Project: Kafka > Issue Type: Sub-task >Reporter: Guozhang Wang > Labels: newbie > > KAFKA-1925 is just a quick-fix of this issue, we need to let consumer to be > able to create separate sockets for the same server for coordinator / broker > roles. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-1935) Consumer should use a separate socket for Coordinator connection
[ https://issues.apache.org/jira/browse/KAFKA-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876537#comment-15876537 ] Colin P. McCabe commented on KAFKA-1935: bq. (By the way, I'm skeptical that sharing connections between different clients is a good idea with the current request handling approach--i.e. head-of-line blocking) Yeah, I agree. I was just bringing it up as an example of people being very concerned about number of sockets. bq. I think this issue is poorly named. The consumer already uses a separate connection to the coordinator and its semantics depend on it doing so. What this issue is fixing is the hack we have used to allow NetworkClient to support multiple connections to the same broker. The solution is probably just to create an actual connectionId (currently the brokerId is used for this). One thing to keep in mind. Currently the NetworkClient is shared between the foreground and background threads in the consumer which has led to us needing to synchronize access through the ConsumerNetworkClient. This is actually sort of tricky to manage, so we've proposed to give the heartbeat thread its own NetworkClient. If we did that, then we wouldn't really need this feature, though I think use cases will probably emerge in the future. Thanks for the clarification. Another good improvement would be allowing multiple messages to be in flight at the same time on the same socket. I've worked on systems where this is possible, and Kafka's correlationId seems to have been added to enable such a feature. I'm not sure if this is a good newbie JIRA or not... what do you think? > Consumer should use a separate socket for Coordinator connection > > > Key: KAFKA-1935 > URL: https://issues.apache.org/jira/browse/KAFKA-1935 > Project: Kafka > Issue Type: Sub-task >Reporter: Guozhang Wang > Labels: newbie > > KAFKA-1925 is just a quick-fix of this issue, we need to let consumer to be > able to create separate sockets for the same server for coordinator / broker > roles. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-1935) Consumer should use a separate socket for Coordinator connection
[ https://issues.apache.org/jira/browse/KAFKA-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876517#comment-15876517 ] Jason Gustafson commented on KAFKA-1935: I think this issue is poorly named. The consumer already uses a separate connection to the coordinator and its semantics depend on it doing so. What this issue is fixing is the hack we have used to allow NetworkClient to support multiple connections to the same broker. The solution is probably just to create an actual connectionId (currently the brokerId is used for this). One thing to keep in mind. Currently the NetworkClient is shared between the foreground and background threads in the consumer which has led to us needing to synchronize access through the ConsumerNetworkClient. This is actually sort of tricky to manage, so we've proposed to give the heartbeat thread its own NetworkClient. If we did that, then we wouldn't really need this feature, though I think use cases will probably emerge in the future. (By the way, I'm skeptical that sharing connections between different clients is a good idea with the current request handling approach--i.e. head-of-line blocking) > Consumer should use a separate socket for Coordinator connection > > > Key: KAFKA-1935 > URL: https://issues.apache.org/jira/browse/KAFKA-1935 > Project: Kafka > Issue Type: Sub-task >Reporter: Guozhang Wang > Labels: newbie > > KAFKA-1925 is just a quick-fix of this issue, we need to let consumer to be > able to create separate sockets for the same server for coordinator / broker > roles. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-1935) Consumer should use a separate socket for Coordinator connection
[ https://issues.apache.org/jira/browse/KAFKA-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876487#comment-15876487 ] Colin P. McCabe commented on KAFKA-1935: In the past, people expressed concerns about the number of sockets that Kafka was using. Someone even proposed sharing the same NetworkClient between producers and consumers in the same process, to avoid opening multiple sockets to the same nodes. If this is a concern, perhaps what we need is to be able to send more than one message at a time on a socket? Another approach would be to change the key to be something more complex than a string, and use something like a class that contained a node id and a string role. > Consumer should use a separate socket for Coordinator connection > > > Key: KAFKA-1935 > URL: https://issues.apache.org/jira/browse/KAFKA-1935 > Project: Kafka > Issue Type: Sub-task >Reporter: Guozhang Wang > Labels: newbie > > KAFKA-1925 is just a quick-fix of this issue, we need to let consumer to be > able to create separate sockets for the same server for coordinator / broker > roles. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-1935) Consumer should use a separate socket for Coordinator connection
[ https://issues.apache.org/jira/browse/KAFKA-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873850#comment-15873850 ] Dipen Patel commented on KAFKA-1935: Hi, I think that this is an interesting first issue for me to work on. Please correct me if I am wrong, but from my understanding the issue we are trying to solve here is ensuring that each coordinator node that we register with the consumer's NetworkClient has a unique Id such that there are no conflicts with the Id's of existing nodes already registered with the consumer's NetworkClient. By ensuring that each coordinator node's Id is unique, we will have successfully mimicked unique sockets for each coordinator node. If that is the issue, then I think that implementing a mechanism to check whether an Id is currently assigned to a node would be an appropriate solution. This mechanism could be implemented by writing a method in the NetworkClient that queries the ClusterConnectionStates (a class containing a Map of all nodes) to check whether the String Id is assigned. We would then have to modify the KafkaClient interface to account for the new method added to the NetworkClient. These changes would allow the AbstractCoordinator class to use its ConsumerNetworkClient field to call the newly implemented method in the NetworkClient. In the case that the Id is not available we increment the Id and repeat the process of checking the Id's uniqueness. Does this proposal sound viable? Thanks > Consumer should use a separate socket for Coordinator connection > > > Key: KAFKA-1935 > URL: https://issues.apache.org/jira/browse/KAFKA-1935 > Project: Kafka > Issue Type: Sub-task >Reporter: Guozhang Wang > Labels: newbie > > KAFKA-1925 is just a quick-fix of this issue, we need to let consumer to be > able to create separate sockets for the same server for coordinator / broker > roles. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
