[jira] [Updated] (KAFKA-2975) The newtorkClient should request a metadata update after it gets an error in the handleResponse()
[ https://issues.apache.org/jira/browse/KAFKA-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-2975: --- Description: Currently in data pipeline, 1) Lets say Mirror Maker requestTimeout is set to 2 min and metadataExpiry is set to 5 min 2) We delete a topic, the Mirror Maker get UNKNOWN_TOPIC_PARTITION and tries torefresh its Metadata. 3) It gets LeaderNotAvailableException, may be because the topic is not created yet. 4) Now its metadata does not have any information about that topic. 5) It will wait for 5 min to do the next refresh. 6) In the mean time the batches sitting in the accumulator will expire and the mirror makers die to avoid data loss. To overcome this we need to refresh the metadata after 3) before the timeout kicks in. Well there is an alternative solution to have the metadataExpiry set to be less then requestTimeout, but this will mean we make more metadataRequest over the wire in normal scenario as well. was: Currently in data pipeline, 1) Lets say Mirror Maker requestTimeout is set to 2 min and metadataExpiry is set to 5 min 2) We delete a topic, the Mirror Maker get UNKNOWN_TOPIC_PARTITION and tries torefresh its Metadata. 3) It gets LeaderNotAvailableException, may be because the topic is not created yet. 4) Now its metadata does not have any information about that topic. 5) It will wait for 5 min to do the next refresh. 6) In the mean tie the batches sitting in the accumulator will expire and the mirror makers die. To overcome this we need to refresh the metadata after 3) before the timeout kicks in. Well there is an alternative solution to have the metadataExpiry set to be less then requestTimeout, but this will mean we make more metadataRequest over the wire in normal scenario as well. > The newtorkClient should request a metadata update after it gets an error in > the handleResponse() > - > > Key: KAFKA-2975 > URL: https://issues.apache.org/jira/browse/KAFKA-2975 > Project: Kafka > Issue Type: Bug > Components: clients >Reporter: Mayuresh Gharat >Assignee: Mayuresh Gharat > > Currently in data pipeline, > 1) Lets say Mirror Maker requestTimeout is set to 2 min and metadataExpiry is > set to 5 min > 2) We delete a topic, the Mirror Maker get UNKNOWN_TOPIC_PARTITION and tries > torefresh its Metadata. > 3) It gets LeaderNotAvailableException, may be because the topic is not > created yet. > 4) Now its metadata does not have any information about that topic. > 5) It will wait for 5 min to do the next refresh. > 6) In the mean time the batches sitting in the accumulator will expire and > the mirror makers die to avoid data loss. > To overcome this we need to refresh the metadata after 3) before the timeout > kicks in. > Well there is an alternative solution to have the metadataExpiry set to be > less then requestTimeout, but this will mean we make more metadataRequest > over the wire in normal scenario as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2975) The newtorkClient should request a metadata update after it gets an error in the handleResponse()
[ https://issues.apache.org/jira/browse/KAFKA-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-2975: --- Description: Currently in data pipeline, 1) Lets say Mirror Maker requestTimeout is set to 2 min and metadataExpiry is set to 5 min 2) We delete a topic, the Mirror Maker get UNKNOWN_TOPIC_PARTITION and tries torefresh its Metadata. 3) It gets LeaderNotAvailableException, may be because the topic is not created yet. 4) Now its metadata does not have any information about that topic. 5) It will wait for 5 min to do the next refresh. 6) In the mean time the batches sitting in the accumulator will expire and the mirror makers die to avoid data loss. To overcome this we need to refresh the metadata after 3). Well there is an alternative solution to have the metadataExpiry set to be less then requestTimeout, but this will mean we make more metadataRequest over the wire in normal scenario as well. was: Currently in data pipeline, 1) Lets say Mirror Maker requestTimeout is set to 2 min and metadataExpiry is set to 5 min 2) We delete a topic, the Mirror Maker get UNKNOWN_TOPIC_PARTITION and tries torefresh its Metadata. 3) It gets LeaderNotAvailableException, may be because the topic is not created yet. 4) Now its metadata does not have any information about that topic. 5) It will wait for 5 min to do the next refresh. 6) In the mean time the batches sitting in the accumulator will expire and the mirror makers die to avoid data loss. To overcome this we need to refresh the metadata after 3) before the timeout kicks in. Well there is an alternative solution to have the metadataExpiry set to be less then requestTimeout, but this will mean we make more metadataRequest over the wire in normal scenario as well. > The newtorkClient should request a metadata update after it gets an error in > the handleResponse() > - > > Key: KAFKA-2975 > URL: https://issues.apache.org/jira/browse/KAFKA-2975 > Project: Kafka > Issue Type: Bug > Components: clients >Reporter: Mayuresh Gharat >Assignee: Mayuresh Gharat > > Currently in data pipeline, > 1) Lets say Mirror Maker requestTimeout is set to 2 min and metadataExpiry is > set to 5 min > 2) We delete a topic, the Mirror Maker get UNKNOWN_TOPIC_PARTITION and tries > torefresh its Metadata. > 3) It gets LeaderNotAvailableException, may be because the topic is not > created yet. > 4) Now its metadata does not have any information about that topic. > 5) It will wait for 5 min to do the next refresh. > 6) In the mean time the batches sitting in the accumulator will expire and > the mirror makers die to avoid data loss. > To overcome this we need to refresh the metadata after 3). > Well there is an alternative solution to have the metadataExpiry set to be > less then requestTimeout, but this will mean we make more metadataRequest > over the wire in normal scenario as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
