Re: New Producer Async - Metadata Fetch Timeout
Thanks Jiangjie. This is helpful. Adding to what you have mentioned, I can think of one more scenario which may not be very rare. Say, the application is rebooted and the Kafka brokers registered in the producer are not reachable ( could be due to network issues or those brokers are actually down ). Since, no metadata is available the send will block. Right? On Wed, May 13, 2015 at 10:51 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: Application will not block on each metadata refresh or metadata is expired. Application will only be blocked when 1. It sends the first message to a topic (only for that single message), or 2. The topic has been deleted from broker thus refreshed metadata loses the topic info (which is pretty rare). So I think the async here might mean a little bit different. It means when you send first message to a topic, you wait till you know the topic exist, after that point it is async. It is very low chance that your application will block on send. If it is then something probably really went wrong and needs immediate attention. Thanks. Jiangjie (Becket) Qin On 5/12/15, 5:08 PM, Rendy Bambang Junior rendy.b.jun...@gmail.com wrote: Thank you for the clarification. I think I agree with Mohit. Sometime blocking on logging is not acceptable by nature of application who uses kafka. Yes it is not blocking when metadata is still available. But application will be blocked once metada is expired. It might be handled by application, by implementing async call when do send() and manage buffer and async timeout internally, but it makes async feature in kafka producer has less meaning. Sorry if my understanding is incorrect. Rendy On May 13, 2015 6:59 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: Send() will only block if the metadata is *not available* for the topic. It won't block if metadata there is stale. The metadata refresh is async to send(). However, if you send the message to a topic for the first time, send() will trigger a metadata refresh and block until it has metadata for that topic. Jiangjie (Becket) Qin On 5/12/15, 12:58 PM, Magnus Edenhill mag...@edenhill.se wrote: I completely agree with Mohit, an application should not have to know or care about producer implementation internals. Given a message and its delivery constraints (produce retry count and timeout) the producer should hide any temporal failures until the message is succesfully delivered, a permanent error is encountered or the constraints are hit. This should also include internal start up sequencing, such as metadata retrieval. 2015-05-12 21:22 GMT+02:00 Mohit Gupta success.mohit.gu...@gmail.com : I could not follow the reasoning behind blocking the send method if the metadata is not up-to-date. Though, I see that it as per design, it requires the metadata to batch the message into appropriate topicPartition queue. Also, if the metadata could not be updated in the specified interval, it throws an exception and the message is not queued to be retried once the brokers are up. Should it not be that messages are buffered in another queue ( up-to a limit ) if the brokers are down and retried later? Is it not a general use case to require producer to be asynchronous in all the scenarios? On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat gharatmayures...@gmail.com wrote: The way it works I suppose is that, the producer will do fetchMetadata, if the last fetched metadata is stale (the refresh interval has expired) or if it is not able to send data to a particular broker in its current metadata (This might happen in some cases like if the leader moves). It cannot produce without having the right metadata. Thanks, Mayuresh On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: That¹s right. Send() will first try to get metadata of a topic, that is a blocking operation. On 5/12/15, 2:48 AM, Rendy Bambang Junior rendy.b.jun...@gmail.com wrote: Hi, sorry if my understanding is incorrect. I am integrating kafka producer with application, when i try to shutdown all kafka broker (preparing for prod env) I notice that 'send' method is blocking. Is new producer fetch metadata not async? Rendy -- -Regards, Mayuresh R. Gharat (862) 250-7125 -- Best Regards, Mohit Gupta -- Best Regards, Mohit Gupta
Re: New Producer Async - Metadata Fetch Timeout
Isn’t the producer part of the application? The metadata is stored in memory. If the application rebooted (process restarted), all the metadata will be gone. Jiangjie (Becket) Qin On 5/13/15, 9:54 AM, Mohit Gupta success.mohit.gu...@gmail.com wrote: I meant the producer. ( i.e. application using the producer api to push messages into kafka ) . On Wed, May 13, 2015 at 10:20 PM, Mayuresh Gharat gharatmayures...@gmail.com wrote: By application rebooting, do you mean you bounce the brokers? Thanks, Mayuresh On Wed, May 13, 2015 at 4:06 AM, Mohit Gupta success.mohit.gu...@gmail.com wrote: Thanks Jiangjie. This is helpful. Adding to what you have mentioned, I can think of one more scenario which may not be very rare. Say, the application is rebooted and the Kafka brokers registered in the producer are not reachable ( could be due to network issues or those brokers are actually down ). Since, no metadata is available the send will block. Right? On Wed, May 13, 2015 at 10:51 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: Application will not block on each metadata refresh or metadata is expired. Application will only be blocked when 1. It sends the first message to a topic (only for that single message), or 2. The topic has been deleted from broker thus refreshed metadata loses the topic info (which is pretty rare). So I think the async here might mean a little bit different. It means when you send first message to a topic, you wait till you know the topic exist, after that point it is async. It is very low chance that your application will block on send. If it is then something probably really went wrong and needs immediate attention. Thanks. Jiangjie (Becket) Qin On 5/12/15, 5:08 PM, Rendy Bambang Junior rendy.b.jun...@gmail.com wrote: Thank you for the clarification. I think I agree with Mohit. Sometime blocking on logging is not acceptable by nature of application who uses kafka. Yes it is not blocking when metadata is still available. But application will be blocked once metada is expired. It might be handled by application, by implementing async call when do send() and manage buffer and async timeout internally, but it makes async feature in kafka producer has less meaning. Sorry if my understanding is incorrect. Rendy On May 13, 2015 6:59 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: Send() will only block if the metadata is *not available* for the topic. It won't block if metadata there is stale. The metadata refresh is async to send(). However, if you send the message to a topic for the first time, send() will trigger a metadata refresh and block until it has metadata for that topic. Jiangjie (Becket) Qin On 5/12/15, 12:58 PM, Magnus Edenhill mag...@edenhill.se wrote: I completely agree with Mohit, an application should not have to know or care about producer implementation internals. Given a message and its delivery constraints (produce retry count and timeout) the producer should hide any temporal failures until the message is succesfully delivered, a permanent error is encountered or the constraints are hit. This should also include internal start up sequencing, such as metadata retrieval. 2015-05-12 21:22 GMT+02:00 Mohit Gupta success.mohit.gu...@gmail.com : I could not follow the reasoning behind blocking the send method if the metadata is not up-to-date. Though, I see that it as per design, it requires the metadata to batch the message into appropriate topicPartition queue. Also, if the metadata could not be updated in the specified interval, it throws an exception and the message is not queued to be retried once the brokers are up. Should it not be that messages are buffered in another queue ( up-to a limit ) if the brokers are down and retried later? Is it not a general use case to require producer to be asynchronous in all the scenarios? On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat gharatmayures...@gmail.com wrote: The way it works I suppose is that, the producer will do fetchMetadata, if the last fetched metadata is stale (the refresh interval has expired) or if it is not able to send data to a particular broker in its current metadata (This might happen in some cases like if the leader moves). It cannot produce without having the right metadata. Thanks, Mayuresh On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: That¹s right. Send() will first try to get metadata of a topic, that is a blocking operation.
Re: New Producer Async - Metadata Fetch Timeout
By application rebooting, do you mean you bounce the brokers? Thanks, Mayuresh On Wed, May 13, 2015 at 4:06 AM, Mohit Gupta success.mohit.gu...@gmail.com wrote: Thanks Jiangjie. This is helpful. Adding to what you have mentioned, I can think of one more scenario which may not be very rare. Say, the application is rebooted and the Kafka brokers registered in the producer are not reachable ( could be due to network issues or those brokers are actually down ). Since, no metadata is available the send will block. Right? On Wed, May 13, 2015 at 10:51 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: Application will not block on each metadata refresh or metadata is expired. Application will only be blocked when 1. It sends the first message to a topic (only for that single message), or 2. The topic has been deleted from broker thus refreshed metadata loses the topic info (which is pretty rare). So I think the async here might mean a little bit different. It means when you send first message to a topic, you wait till you know the topic exist, after that point it is async. It is very low chance that your application will block on send. If it is then something probably really went wrong and needs immediate attention. Thanks. Jiangjie (Becket) Qin On 5/12/15, 5:08 PM, Rendy Bambang Junior rendy.b.jun...@gmail.com wrote: Thank you for the clarification. I think I agree with Mohit. Sometime blocking on logging is not acceptable by nature of application who uses kafka. Yes it is not blocking when metadata is still available. But application will be blocked once metada is expired. It might be handled by application, by implementing async call when do send() and manage buffer and async timeout internally, but it makes async feature in kafka producer has less meaning. Sorry if my understanding is incorrect. Rendy On May 13, 2015 6:59 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: Send() will only block if the metadata is *not available* for the topic. It won't block if metadata there is stale. The metadata refresh is async to send(). However, if you send the message to a topic for the first time, send() will trigger a metadata refresh and block until it has metadata for that topic. Jiangjie (Becket) Qin On 5/12/15, 12:58 PM, Magnus Edenhill mag...@edenhill.se wrote: I completely agree with Mohit, an application should not have to know or care about producer implementation internals. Given a message and its delivery constraints (produce retry count and timeout) the producer should hide any temporal failures until the message is succesfully delivered, a permanent error is encountered or the constraints are hit. This should also include internal start up sequencing, such as metadata retrieval. 2015-05-12 21:22 GMT+02:00 Mohit Gupta success.mohit.gu...@gmail.com : I could not follow the reasoning behind blocking the send method if the metadata is not up-to-date. Though, I see that it as per design, it requires the metadata to batch the message into appropriate topicPartition queue. Also, if the metadata could not be updated in the specified interval, it throws an exception and the message is not queued to be retried once the brokers are up. Should it not be that messages are buffered in another queue ( up-to a limit ) if the brokers are down and retried later? Is it not a general use case to require producer to be asynchronous in all the scenarios? On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat gharatmayures...@gmail.com wrote: The way it works I suppose is that, the producer will do fetchMetadata, if the last fetched metadata is stale (the refresh interval has expired) or if it is not able to send data to a particular broker in its current metadata (This might happen in some cases like if the leader moves). It cannot produce without having the right metadata. Thanks, Mayuresh On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: That¹s right. Send() will first try to get metadata of a topic, that is a blocking operation. On 5/12/15, 2:48 AM, Rendy Bambang Junior rendy.b.jun...@gmail.com wrote: Hi, sorry if my understanding is incorrect. I am integrating kafka producer with application, when i try to shutdown all kafka broker (preparing for prod env) I notice that 'send' method is blocking. Is new producer fetch metadata not async? Rendy -- -Regards, Mayuresh R. Gharat (862) 250-7125 -- Best Regards, Mohit Gupta -- Best
Re: New Producer Async - Metadata Fetch Timeout
I meant the producer. ( i.e. application using the producer api to push messages into kafka ) . On Wed, May 13, 2015 at 10:20 PM, Mayuresh Gharat gharatmayures...@gmail.com wrote: By application rebooting, do you mean you bounce the brokers? Thanks, Mayuresh On Wed, May 13, 2015 at 4:06 AM, Mohit Gupta success.mohit.gu...@gmail.com wrote: Thanks Jiangjie. This is helpful. Adding to what you have mentioned, I can think of one more scenario which may not be very rare. Say, the application is rebooted and the Kafka brokers registered in the producer are not reachable ( could be due to network issues or those brokers are actually down ). Since, no metadata is available the send will block. Right? On Wed, May 13, 2015 at 10:51 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: Application will not block on each metadata refresh or metadata is expired. Application will only be blocked when 1. It sends the first message to a topic (only for that single message), or 2. The topic has been deleted from broker thus refreshed metadata loses the topic info (which is pretty rare). So I think the async here might mean a little bit different. It means when you send first message to a topic, you wait till you know the topic exist, after that point it is async. It is very low chance that your application will block on send. If it is then something probably really went wrong and needs immediate attention. Thanks. Jiangjie (Becket) Qin On 5/12/15, 5:08 PM, Rendy Bambang Junior rendy.b.jun...@gmail.com wrote: Thank you for the clarification. I think I agree with Mohit. Sometime blocking on logging is not acceptable by nature of application who uses kafka. Yes it is not blocking when metadata is still available. But application will be blocked once metada is expired. It might be handled by application, by implementing async call when do send() and manage buffer and async timeout internally, but it makes async feature in kafka producer has less meaning. Sorry if my understanding is incorrect. Rendy On May 13, 2015 6:59 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: Send() will only block if the metadata is *not available* for the topic. It won't block if metadata there is stale. The metadata refresh is async to send(). However, if you send the message to a topic for the first time, send() will trigger a metadata refresh and block until it has metadata for that topic. Jiangjie (Becket) Qin On 5/12/15, 12:58 PM, Magnus Edenhill mag...@edenhill.se wrote: I completely agree with Mohit, an application should not have to know or care about producer implementation internals. Given a message and its delivery constraints (produce retry count and timeout) the producer should hide any temporal failures until the message is succesfully delivered, a permanent error is encountered or the constraints are hit. This should also include internal start up sequencing, such as metadata retrieval. 2015-05-12 21:22 GMT+02:00 Mohit Gupta success.mohit.gu...@gmail.com : I could not follow the reasoning behind blocking the send method if the metadata is not up-to-date. Though, I see that it as per design, it requires the metadata to batch the message into appropriate topicPartition queue. Also, if the metadata could not be updated in the specified interval, it throws an exception and the message is not queued to be retried once the brokers are up. Should it not be that messages are buffered in another queue ( up-to a limit ) if the brokers are down and retried later? Is it not a general use case to require producer to be asynchronous in all the scenarios? On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat gharatmayures...@gmail.com wrote: The way it works I suppose is that, the producer will do fetchMetadata, if the last fetched metadata is stale (the refresh interval has expired) or if it is not able to send data to a particular broker in its current metadata (This might happen in some cases like if the leader moves). It cannot produce without having the right metadata. Thanks, Mayuresh On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: That¹s right. Send() will first try to get metadata of a topic, that is a blocking operation. On 5/12/15, 2:48 AM, Rendy Bambang Junior rendy.b.jun...@gmail.com wrote: Hi, sorry if my understanding is incorrect. I am integrating kafka producer with application, when i try to shutdown
Re: New Producer Async - Metadata Fetch Timeout
The way it works I suppose is that, the producer will do fetchMetadata, if the last fetched metadata is stale (the refresh interval has expired) or if it is not able to send data to a particular broker in its current metadata (This might happen in some cases like if the leader moves). It cannot produce without having the right metadata. Thanks, Mayuresh On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: That¹s right. Send() will first try to get metadata of a topic, that is a blocking operation. On 5/12/15, 2:48 AM, Rendy Bambang Junior rendy.b.jun...@gmail.com wrote: Hi, sorry if my understanding is incorrect. I am integrating kafka producer with application, when i try to shutdown all kafka broker (preparing for prod env) I notice that 'send' method is blocking. Is new producer fetch metadata not async? Rendy -- -Regards, Mayuresh R. Gharat (862) 250-7125
Re: New Producer Async - Metadata Fetch Timeout
I could not follow the reasoning behind blocking the send method if the metadata is not up-to-date. Though, I see that it as per design, it requires the metadata to batch the message into appropriate topicPartition queue. Also, if the metadata could not be updated in the specified interval, it throws an exception and the message is not queued to be retried once the brokers are up. Should it not be that messages are buffered in another queue ( up-to a limit ) if the brokers are down and retried later? Is it not a general use case to require producer to be asynchronous in all the scenarios? On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat gharatmayures...@gmail.com wrote: The way it works I suppose is that, the producer will do fetchMetadata, if the last fetched metadata is stale (the refresh interval has expired) or if it is not able to send data to a particular broker in its current metadata (This might happen in some cases like if the leader moves). It cannot produce without having the right metadata. Thanks, Mayuresh On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: That¹s right. Send() will first try to get metadata of a topic, that is a blocking operation. On 5/12/15, 2:48 AM, Rendy Bambang Junior rendy.b.jun...@gmail.com wrote: Hi, sorry if my understanding is incorrect. I am integrating kafka producer with application, when i try to shutdown all kafka broker (preparing for prod env) I notice that 'send' method is blocking. Is new producer fetch metadata not async? Rendy -- -Regards, Mayuresh R. Gharat (862) 250-7125 -- Best Regards, Mohit Gupta
Re: New Producer Async - Metadata Fetch Timeout
That¹s right. Send() will first try to get metadata of a topic, that is a blocking operation. On 5/12/15, 2:48 AM, Rendy Bambang Junior rendy.b.jun...@gmail.com wrote: Hi, sorry if my understanding is incorrect. I am integrating kafka producer with application, when i try to shutdown all kafka broker (preparing for prod env) I notice that 'send' method is blocking. Is new producer fetch metadata not async? Rendy
Re: New Producer Async - Metadata Fetch Timeout
Send() will only block if the metadata is *not available* for the topic. It won’t block if metadata there is stale. The metadata refresh is async to send(). However, if you send the message to a topic for the first time, send() will trigger a metadata refresh and block until it has metadata for that topic. Jiangjie (Becket) Qin On 5/12/15, 12:58 PM, Magnus Edenhill mag...@edenhill.se wrote: I completely agree with Mohit, an application should not have to know or care about producer implementation internals. Given a message and its delivery constraints (produce retry count and timeout) the producer should hide any temporal failures until the message is succesfully delivered, a permanent error is encountered or the constraints are hit. This should also include internal start up sequencing, such as metadata retrieval. 2015-05-12 21:22 GMT+02:00 Mohit Gupta success.mohit.gu...@gmail.com: I could not follow the reasoning behind blocking the send method if the metadata is not up-to-date. Though, I see that it as per design, it requires the metadata to batch the message into appropriate topicPartition queue. Also, if the metadata could not be updated in the specified interval, it throws an exception and the message is not queued to be retried once the brokers are up. Should it not be that messages are buffered in another queue ( up-to a limit ) if the brokers are down and retried later? Is it not a general use case to require producer to be asynchronous in all the scenarios? On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat gharatmayures...@gmail.com wrote: The way it works I suppose is that, the producer will do fetchMetadata, if the last fetched metadata is stale (the refresh interval has expired) or if it is not able to send data to a particular broker in its current metadata (This might happen in some cases like if the leader moves). It cannot produce without having the right metadata. Thanks, Mayuresh On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: That¹s right. Send() will first try to get metadata of a topic, that is a blocking operation. On 5/12/15, 2:48 AM, Rendy Bambang Junior rendy.b.jun...@gmail.com wrote: Hi, sorry if my understanding is incorrect. I am integrating kafka producer with application, when i try to shutdown all kafka broker (preparing for prod env) I notice that 'send' method is blocking. Is new producer fetch metadata not async? Rendy -- -Regards, Mayuresh R. Gharat (862) 250-7125 -- Best Regards, Mohit Gupta
