Re: AVRO Schema with optional field ("type": "record")

[Jacob Sheck]

Can you use a union with null?  This would be the IDL exmple.

record RecordConfig {
...
}

 union {null, RecordConfig} record = null;

On Fri, Oct 19, 2018 at 2:44 PM Mina Aslani  wrote:

> Hi,
>
> I am adding AVRO schema to the kafka messages, however, I would like to
> know how I can make a field of "type": "record" optional.
> *Note*: "default":null does not help.
>
> Any idea? Can you elaborate the solution/workaround with an example please?
>
> Best regards,
> Mina
>

Re: Kafka commit interval

[pradeep s]

Thanks Matthias

On Fri, Oct 19, 2018 at 11:25 AM Matthias J. Sax 
wrote:

> There is not upper limit.
>
> And yes, you are right about rebalancing. This would be an issue and
> yes, you can use the rebalance listener to address it (it's the purpose
> of the rebalance listener to be used for cases like this).
>
> -Matthias
>
>
> On 10/16/18 2:19 PM, pradeep s wrote:
> > Hi,
> > I have a usecase to stream messages from Kafka and buffer it in memory
> till
> > a message count is reached and then write these to output file . I am
> using
> > manual commit . I have a question on whats the maximum time i can wait
> > after consuming the message and till we commit back to Kafka . Is there a
> > upper limit for this ? Is consumer rebalance the only issue while waiting
> > long periods, then can i write the contents using a rebalance listener
> and
> > commit at that point .
> > Thanks
> > Pradeep
> >
>
>

AVRO Schema with optional field ("type": "record")

[Mina Aslani]

Hi,

I am adding AVRO schema to the kafka messages, however, I would like to
know how I can make a field of "type": "record" optional.
*Note*: "default":null does not help.

Any idea? Can you elaborate the solution/workaround with an example please?

Best regards,
Mina

Re: Kafka commit interval

[Matthias J. Sax]

There is not upper limit.

And yes, you are right about rebalancing. This would be an issue and
yes, you can use the rebalance listener to address it (it's the purpose
of the rebalance listener to be used for cases like this).

-Matthias


On 10/16/18 2:19 PM, pradeep s wrote:
> Hi,
> I have a usecase to stream messages from Kafka and buffer it in memory till
> a message count is reached and then write these to output file . I am using
> manual commit . I have a question on whats the maximum time i can wait
> after consuming the message and till we commit back to Kafka . Is there a
> upper limit for this ? Is consumer rebalance the only issue while waiting
> long periods, then can i write the contents using a rebalance listener and
> commit at that point .
> Thanks
> Pradeep
> 



signature.asc
Description: OpenPGP digital signature

Re: Kafka Streams, when is considered processed?

[Matthias J. Sax]

A message is considered processed, if all state updates are done and all
output messages are written.

Note, that this notion of "processed" is based on sub-topologies, but
not the full topology.

Hope this helps.


-Matthias

On 10/18/18 4:28 AM, Tobias Johansson wrote:
> Hi,
> 
> 
> I can't find the following in the documentation.
> 
> 
> In a Kafka Streams application, I wonder for when a message is considered 
> processed.
> 
> 
> For example for exactly-once configured, we are promised exactly-once 
> processing. But when is it considered processed, directly after consumption?
> 
> 
> Regards, Tobias
> 
> 
> Tobias Johansson
> Java Developer
> 
> NetEnt | Better Gaming(tm)
> T: +46 73 987 28 63, M: +46 73 987 28 
> 63
> tobias.johans...@netent.com, 
> www.netent.com
> Address: NetEnt AB (Publ), Vasagatan 16, 111 20, Stockholm, SE
> This email and any attachments are confidential and may be legally privileged 
> and protected by copyright. If you are not the intended recipient of this 
> email you should not copy it or disclose its contents to anyone. If you have 
> received this email in error, please notify the sender immediately and delete 
> the email. Views or opinions in this email are solely those of the author. 
> Unencrypted Internet communications are not secure and the sender does not 
> accept responsibility for interception of this message by third parties. This 
> communication is not intended to form a binding contract unless expressly 
> indicated to the contrary and properly authorized. The recipient should scan 
> this email and any attachments for the presence of viruses. The sender 
> accepts no liability for any viruses transmitted in this email.
> 



signature.asc
Description: OpenPGP digital signature

Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

Since this has gone quiet, could I prequest 1 more vote here - if anyone
thinks it's worth doing?

Thanks,



On Wed, 3 Oct 2018 at 16:14, M. Manna  wrote:

> Thanks for reminding me about the "Binding" vote Bill. I remember some
> people with non-binding vote, so jumped the gun a bit too early.
> We will wait for 2 more as you stated.
>
> Regards,
>
> On Wed, 3 Oct 2018 at 16:07, Bill Bejeck  wrote:
>
>> +1 from me.
>>
>> As for closing the vote, it needs to be open for a minimum of 72 and
>> requires three binding +1 votes.  Additionally, there needs more +1
>> binding
>> votes than -1 votes.  The description for the lazy majority vote process
>> is
>> described here
>> https://cwiki.apache.org/confluence/display/KAFKA/Bylaws#Bylaws-Approvals
>> .
>>
>> I haven't been tracking the vote results, but from what I can see in the
>> voting thread, this KIP still requires two more +1 binding votes.
>>
>> HTH,
>> BIll
>>
>> On Wed, Oct 3, 2018 at 8:58 AM M. Manna  wrote:
>>
>> > Since this has been open for a while, I am assuming that it's good to
>> go?
>> >
>> > if so, I will update the KIP page - and start coding this. I would
>> prefer
>> > re-using existing tests written for DefaultPartitioner, so that we don't
>> > need to write new tests.
>> >
>> > Regards,
>> >
>> > On Sun, 30 Sep 2018 at 19:34, Matthias J. Sax 
>> > wrote:
>> >
>> > > +1 (binding)
>> > >
>> > > @Abhimanyu: can you please update the table in
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>> > > and add a link to the KIP. Thanks.
>> > >
>> > > -Matthias
>> > >
>> > > On 9/4/18 9:56 PM, Abhimanyu Nagrath wrote:
>> > > > +1
>> > > >
>> > > > On Wed, Sep 5, 2018 at 2:39 AM Magesh Nandakumar <
>> mage...@confluent.io
>> > >
>> > > > wrote:
>> > > >
>> > > >> +1 ( non-binding)
>> > > >>
>> > > >> On Tue, Sep 4, 2018 at 3:39 AM M. Manna 
>> wrote:
>> > > >>
>> > > >>> Hello,
>> > > >>>
>> > > >>> I have made necessary changes as per the original discussion
>> thread,
>> > > and
>> > > >>> would like to put it for votes.
>> > > >>>
>> > > >>> Thank you very much for your suggestion and guidance so far.
>> > > >>>
>> > > >>> Regards,
>> > > >>>
>> > > >>
>> > > >
>> > >
>> > >
>> >
>>
>

Re: Visual tool for kafka?

[Peter Birksmith]

What about Landoop?
https://www.landoop.com/
 
Cheers
 
Peter Birksmith
Senior Analyst
Customer Labs
Phone: +61 2 9292 9167 Ext. 29167
Fax: +61 (0)2 9292 
Email: peter.birksm...@iag.com.au
Tower Two, Darling Park,
201 Sussex Street, Sydney NSW 2000
 

 
We help make your world a safer place.
 

On 19/10/18, 6:08 pm, "Peter Bukowinski"  wrote:

https://github.com/yahoo/kafka-manager 


This does all that, and has admin features (which you can disable) that 
allow you to change or create topics, do partition reassignment and preferred 
leader election.

—
Peter

> On Oct 18, 2018, at 11:52 PM, 1095193...@qq.com wrote:
> 
> Hi 
>   I need a Visual tool for kafka. For example, A Web UI can display the 
detail of each topics、the offset of each consumer. Has any recommended visual 
tools?
> 
> 
> 
> 1095193...@qq.com




_

The information transmitted in this message and its attachments (if any) is 
intended 
only for the person or entity to which it is addressed.
The message may contain confidential and/or privileged material. Any review, 
retransmission, dissemination or other use of, or taking of any action in 
reliance 
upon this information, by persons or entities other than the intended recipient 
is 
prohibited.

If you have received this in error, please contact the sender and delete this 
e-mail 
and associated material from any computer.

The intended recipient of this e-mail may only use, reproduce, disclose or 
distribute 
the information contained in this e-mail and any attached files, with the 
permission 
of the sender.

This message has been scanned for viruses.
_

Re: Visual tool for kafka?

[Peter Bukowinski]

https://github.com/yahoo/kafka-manager 

This does all that, and has admin features (which you can disable) that allow 
you to change or create topics, do partition reassignment and preferred leader 
election.

—
Peter

> On Oct 18, 2018, at 11:52 PM, 1095193...@qq.com wrote:
> 
> Hi 
>   I need a Visual tool for kafka. For example, A Web UI can display the 
> detail of each topics、the offset of each consumer. Has any recommended visual 
> tools?
> 
> 
> 
> 1095193...@qq.com

Visual tool for kafka?

[1095193...@qq.com]

Hi 
   I need a Visual tool for kafka. For example, A Web UI can display the detail 
of each topics、the offset of each consumer. Has any recommended visual tools?



1095193...@qq.com

offsets.topic.replication.factor vs default.replication.factor

[Raghav]

Hi

We have a 3 node Kafka Brokers setup.

Our current value of default.replication.factor is 2.

What should be the recommended value of offsets.topic.replication.factor ?

Please advise as we are not completely sure
about offsets.topic.replication.factor ?

Thanks for your help.

R