Re: [VOTE] KIP-749: Add --files and --file-separator options to the ConsoleProducer

[wenbing shen]

Thanks for your comments Ran.

`kafka-console-producer.sh --broker-list localhost:9092 --topic
my_topic <(cat file1 file2)`
Does this command work properly?

You are right, with the help of shell, we can also complete multi-file
reading.
In the windows environment, perhaps we will also use bat to complete
multi-file reading.
Adding options from the producer command line tool to support this feature
is also to make it easier to use kafka.

Thanks,

Wenbing

Ran Lupovich  于2021年6月7日周一 上午11:51写道：

>  kafka-console-producer.sh --broker-list localhost:9092 --topic
> my_topic <(cat file1 file2)
>
> בתאריך יום ב׳, 7 ביוני 2021, 05:05, מאת Ran Lupovich ‏<
> ranlupov...@gmail.com
> >:
>
> > kafka-console-producer.sh --broker-list localhost:9092 --topic my_topic
> < my_file.txt
> >
> >
> > ?
> >
> >
> > בתאריך יום א׳, 6 ביוני 2021, 22:43, מאת Boyang Chen ‏<
> > reluctanthero...@gmail.com>:
> >
> >> Thanks for the KIP Wenbing. I have a couple of questions:
> >>
> >> 1. Could you add more descriptions into the motivation section, as of
> when
> >> the new options are favored and what's the current workaround?
> >> 2. Could you separate out the public interface and proposed changes
> >> sections, such that the new options are put in the former section and
> the
> >> expected changes are in latter.
> >> 3. I don't see a necessity to add support for different file separators,
> >> could you elaborate?
> >>
> >> Boyang
> >>
> >> On Sat, Jun 5, 2021 at 5:16 PM wenbing shen 
> >> wrote:
> >>
> >> > Hi all,
> >> >
> >> > I'd like to start a vote on KIP-749 to add two options (--files and
> >> > --files-separator) to ConsoleProducer.
> >> >
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-749:+Add+--files+and+--file-separator+options+to+the+ConsoleProducer
> >> >
> >> > Thanks,
> >> >
> >> > Wenbing
> >> >
> >>
> >>
>

Re: [VOTE] KIP-618: Exactly-once support for source connectors

[Dongjin Lee]

+1 (non-binding).

Thanks,
Dongjin

On Fri, Jun 4, 2021 at 2:35 PM Ryanne Dolan  wrote:

> +1 (non-binding)
>
> On Thu, Jun 3, 2021, 10:23 AM Chris Egerton 
> wrote:
>
> > Hi all,
> >
> > I'd like to call for a vote on KIP-618, which adds support for
> exactly-once
> > delivery guarantees for source connectors in the Kafka Connect framework.
> >
> > I suspect there might be a little more discussion to be had but with the
> > KIP freeze deadline approaching, I wanted to give anyone following along
> > the chance to cast a +1 as soon as they feel that we've gotten to a
> > reasonable state.
> >
> > The KIP:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors
> >
> > The discussion thread:
> >
> >
> https://mail-archives.apache.org/mod_mbox/kafka-dev/202005.mbox/%3CCAMdOrUX-VK5OSB3OOdJNXW_YKEJH9FjQZ4eyzr2GXUhSeDnF3Q%40mail.gmail.com%3E
> >
> > Cheers,
> >
> > Chris
> >
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [VOTE] KIP-726: Make the "cooperative-sticky, range" as the default assignor

[Dongjin Lee]

+1 (non-binding)

Best,
Dongjin

On Fri, Jun 4, 2021 at 9:23 AM Guozhang Wang  wrote:

> Thanks Luke, I'm +1 on this proposal.
>
> Guozhang
>
> On Wed, Jun 2, 2021 at 8:16 PM Luke Chen  wrote:
>
> > Hi all,
> > I'd like to call for a vote on KIP-726: Make the "cooperative-sticky,
> > range" as the default assignor.
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048248
> >
> > This KIP is still waiting for the prerequisite stories to get completed,
> > but it should be soon. Hopefully this can be put into V3.0 since
> > cooperative rebalancing is a major
> > improvement to the life of a consumer group (and its operator).
> >
> > The discussion thread can be found here:
> >
> >
> https://lists.apache.org/thread.html/ref63417ea84a58c9068ea025b3ad38ca2cc340f5818ac07cd83452b7%40%3Cdev.kafka.apache.org%3E
> >
> > Thank you.
> > Luke
> >
>
>
> --
> -- Guozhang
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [VOTE] KIP-749: Add --files and --file-separator options to the ConsoleProducer

[wenbing shen]

Thanks for your questions and comments Boyang.

1.Regarding the first point you mentioned:
If we have some offline log files, we need to enter data into kafka for
analysis and use in subsequent processes.
This situation may not be frequent, but it does exist.
After this KIP, these data can be written to kafka quickly and easily
through the ConsoleProducer.

Supporting multiple file reading through options, rather than the
assistance of complex bat and shell scripts, would be a good choice.
This is also an improvement that makes Kafka easier to use.

2.Regarding the description of the KIP content in the first and second
points, I have updated the KIP.

3.Regarding your third question, I agree with you. The comma separator
conforms to the habit of most users, and this habit does not change often.
Therefore, we will use a comma as the default separator and let users
follow this rule.

Thanks

Wenbing

Boyang Chen  于2021年6月7日周一 上午3:43写道：

> Thanks for the KIP Wenbing. I have a couple of questions:
>
> 1. Could you add more descriptions into the motivation section, as of when
> the new options are favored and what's the current workaround?
> 2. Could you separate out the public interface and proposed changes
> sections, such that the new options are put in the former section and the
> expected changes are in latter.
> 3. I don't see a necessity to add support for different file separators,
> could you elaborate?
>
> Boyang
>
> On Sat, Jun 5, 2021 at 5:16 PM wenbing shen 
> wrote:
>
> > Hi all,
> >
> > I'd like to start a vote on KIP-749 to add two options (--files and
> > --files-separator) to ConsoleProducer.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-749:+Add+--files+and+--file-separator+options+to+the+ConsoleProducer
> >
> > Thanks,
> >
> > Wenbing
> >
>

Re: [VOTE] KIP-749: Add --files and --file-separator options to the ConsoleProducer

[Ran Lupovich]

 kafka-console-producer.sh --broker-list localhost:9092 --topic
my_topic <(cat file1 file2)

בתאריך יום ב׳, 7 ביוני 2021, 05:05, מאת Ran Lupovich ‏:

> kafka-console-producer.sh --broker-list localhost:9092 --topic my_topic < 
> my_file.txt
>
>
> ?
>
>
> בתאריך יום א׳, 6 ביוני 2021, 22:43, מאת Boyang Chen ‏<
> reluctanthero...@gmail.com>:
>
>> Thanks for the KIP Wenbing. I have a couple of questions:
>>
>> 1. Could you add more descriptions into the motivation section, as of when
>> the new options are favored and what's the current workaround?
>> 2. Could you separate out the public interface and proposed changes
>> sections, such that the new options are put in the former section and the
>> expected changes are in latter.
>> 3. I don't see a necessity to add support for different file separators,
>> could you elaborate?
>>
>> Boyang
>>
>> On Sat, Jun 5, 2021 at 5:16 PM wenbing shen 
>> wrote:
>>
>> > Hi all,
>> >
>> > I'd like to start a vote on KIP-749 to add two options (--files and
>> > --files-separator) to ConsoleProducer.
>> >
>> >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-749:+Add+--files+and+--file-separator+options+to+the+ConsoleProducer
>> >
>> > Thanks,
>> >
>> > Wenbing
>> >
>>
>>

Re: [VOTE] KIP-749: Add --files and --file-separator options to the ConsoleProducer

[Ran Lupovich]

kafka-console-producer.sh --broker-list localhost:9092 --topic
my_topic < my_file.txt


?


בתאריך יום א׳, 6 ביוני 2021, 22:43, מאת Boyang Chen ‏<
reluctanthero...@gmail.com>:

> Thanks for the KIP Wenbing. I have a couple of questions:
>
> 1. Could you add more descriptions into the motivation section, as of when
> the new options are favored and what's the current workaround?
> 2. Could you separate out the public interface and proposed changes
> sections, such that the new options are put in the former section and the
> expected changes are in latter.
> 3. I don't see a necessity to add support for different file separators,
> could you elaborate?
>
> Boyang
>
> On Sat, Jun 5, 2021 at 5:16 PM wenbing shen 
> wrote:
>
> > Hi all,
> >
> > I'd like to start a vote on KIP-749 to add two options (--files and
> > --files-separator) to ConsoleProducer.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-749:+Add+--files+and+--file-separator+options+to+the+ConsoleProducer
> >
> > Thanks,
> >
> > Wenbing
> >
>

Re: [VOTE] KIP-749: Add --files and --file-separator options to the ConsoleProducer

[Boyang Chen]

Thanks for the KIP Wenbing. I have a couple of questions:

1. Could you add more descriptions into the motivation section, as of when
the new options are favored and what's the current workaround?
2. Could you separate out the public interface and proposed changes
sections, such that the new options are put in the former section and the
expected changes are in latter.
3. I don't see a necessity to add support for different file separators,
could you elaborate?

Boyang

On Sat, Jun 5, 2021 at 5:16 PM wenbing shen 
wrote:

> Hi all,
>
> I'd like to start a vote on KIP-749 to add two options (--files and
> --files-separator) to ConsoleProducer.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-749:+Add+--files+and+--file-separator+options+to+the+ConsoleProducer
>
> Thanks,
>
> Wenbing
>

[jira] [Created] (KAFKA-12901) Metadata not updated after broker restart.

[Suriya Vijayaraghavan (Jira)]

Suriya Vijayaraghavan created KAFKA-12901:
-

 Summary: Metadata not updated after broker restart.
 Key: KAFKA-12901
 URL: https://issues.apache.org/jira/browse/KAFKA-12901
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.8.0
Reporter: Suriya Vijayaraghavan



We upgraded to version 2.8 from 2.7. After monitoring for few weeks we upgraded 
in our production setup (as we didn't enable Kraft we went ahead), we faced 
TimeoutException in our clients after few weeks in our production setup. We 
tried to list all active brokers using admin client API, all brokers were 
listed properly. So we logged into that broker and tried to do a describe topic 
with localhost as bootstrap-server, but we got timeout as there.

When checking the logs, we noticed a Shutdown print from kafka-shutdown-hook
thread (zookeeper session timed out and we had three retry failures). But the 
controlled shutdown got failed (got unknown server error response from the 
controller), and proceeded to unclean shutdown. Still the process didn't get 
quit but the process didnt process any other operation as well.  And this did 
not remove the broker from alive status for hours (able to see this broker in 
list of brokers) and our clients were still trying to contact this broker and 
failing with timeout exception. So we tried restarting the problematic broker, 
but we faced unknown topic or partition issue in our client after the restart. 
We noticed that metadata was not loaded. So we had to restart our controller. 
And after restarting the controller everthing got back to normal.

So how metadata loading is handled? Is there any alternative ways for us to 
automate monitoring for metadata update? 




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-390: Support Compression Level (rebooted)

[Ryanne Dolan]

+1 (non-binding), thanks!

Ryanne

On Sat, Jun 5, 2021, 4:37 PM Dongjin Lee  wrote:

> Hi all,
>
> I'd like to open a voting thread for KIP-390: Support Compression Level
> (rebooted):
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level
>
> Best,
> Dongjin
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck:
> speakerdeck.com/dongjin
> *
>

Re: [VOTE] KIP-752: Support --bootstrap-server in ReplicaVerificationTool

[Luke Chen]

Hi Dongjin,
Thanks for the KIP.
+1 (non-binding)

Thanks.
Luke

On Sun, Jun 6, 2021 at 5:38 AM Dongjin Lee  wrote:

> Hi all,
>
> I'd like to call for a vote on KIP-752: Support --bootstrap-server in
> ReplicaVerificationTool:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-752%3A+Support+--bootstrap-server+in+ReplicaVerificationTool
>
> Best,
> Dongjin
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck:
> speakerdeck.com/dongjin
> *
>

Re: [VOTE] KIP-749: Add --files and --file-separator options to the ConsoleProducer

[Luke Chen]

Hi Wenbing,
Thanks for the KIP!
+1 (non-binding)

Thanks.
Luke

On Sun, Jun 6, 2021 at 8:16 AM wenbing shen 
wrote:

> Hi all,
>
> I'd like to start a vote on KIP-749 to add two options (--files and
> --files-separator) to ConsoleProducer.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-749:+Add+--files+and+--file-separator+options+to+the+ConsoleProducer
>
> Thanks,
>
> Wenbing
>