Kafka-Mulesoft integration

[Tanvi Aggarwal]

Hi Team,

I am trying to integrate Mulesoft and Kafka.
Getting below error while running the project --
org.apache.kafka.common.errors.ApiException: The configured groupId is
invalid


And there is no standard field to set groupId in Mulesoft.
Can you please help if it has something to do with the versions to be used.

Thanks,
Tanvi

Re: Kafka retention hooks

[Radoslaw Gruchalski]

John,

AFAIK not, however, this was suggested as part of the following JIRA:
https://issues.apache.org/jira/browse/KAFKA-3726
Feel free to upvote.

–
Best regards,
Radek Gruchalski
ra...@gruchalski.com


On October 4, 2016 at 12:17:08 AM, John Vines (jvi...@gmail.com) wrote:

Obligatory sorry if I missed it, but I looked at couldn't find anything.

Does Kafka support any sort of actions to perform when an event gets
truncated due to the retention policy?

Background: I'm looking into using kafka to augment our existing rigid data
flow by using it as a messaging system, but I'm concerned about systems
being unavailable, resulting in persistent data not being cleaned up. I
accept the possibility of missed messages with the exception of a 'missed
cleanup', so I was hoping for a hook available to use to ensure that at
least one action was always available.

Thanks
John

Kafka retention hooks

[John Vines]

Obligatory sorry if I missed it, but I looked at couldn't find anything.

Does Kafka support any sort of actions to perform when an event gets
truncated due to the retention policy?

Background: I'm looking into using kafka to augment our existing rigid data
flow by using it as a messaging system, but I'm concerned about systems
being unavailable, resulting in persistent data not being cleaned up. I
accept the possibility of missed messages with the exception of a 'missed
cleanup', so I was hoping for a hook available to use to ensure that at
least one action was always available.

Thanks
John

What happens after broker reaches disk write limit

[Jiayue Zhang (Bravo)]

Hi,

My Kafka version is 0.8.2.2 Replica factor is
2. auto.leader.rebalance.enable=true

I stopped a broker in my cluster. After a few minutes I started this
broker. The broker was busy catching up huge lag and reached 120MB/s disk
write limit. Additionally there are 23 partitions whose only undead replica
lives on this broker. So shortly after this broker was up, it became the
leader of the 23 partitions. After about 10 minutes this broker was up, all
brokers' network and message in started to fall. Messages in dropped to
close to zero after another 5 minutes. It was only when I stopped this
broker that producers recovered to send messages to cluster again.

1. When a broker reaches 120MB/s disk write limit, how will it affect other
brokers and producers?
2. What caused other brokers' or producers' problem?
3. If broker 3 became leaders of NO partitions after I restarted it, would
it still cause problem even if it reached disk write limit when it was busy
catching up as follows?
4. If question 3's answer is 'yes', what should I do to safely restart a
broker with huge traffic?
5. Does auto.leader.rebalance.enable=true do anything harmful?

Thanks

Re: SSL Kafka 0.9.0.1

[Harsha Chintalapani]

Shri,
  SSL in 0.9.0.1 is not beta and can be used in production. If you want
to put authorizer on top of SSL to enable ACLs for clients and topics
 that's possible too.

Thanks,
Harsha

On Mon, Oct 3, 2016 at 8:30 AM Shrikant Patel  wrote:

> We are are 0.9.0.1 and want to use SSL for ACL and securing communication
> between borker, producer and consumer.
>
> Was \ Is the SSL based ACL in beta for this version of Kafka???
>
>
> We don't want upgrade to 0.10.x unless it absolutely needed.
>
> Thanks,
> Shri
> __
> Shrikant Patel   |   PDX-NHIN
> Enterprise Architecture Team
> Asserting the Role of Pharmacy in Healthcare  www.pdxinc.com<
> http://www.pdxinc.com/>
> main 817.246.6760 | ext 4302
> 101 Jim Wright Freeway South, Suite 200, Fort Worth, Texas 76108-2202<
> http://maps.google.com/maps?q=PDX,+Inc.=en=32.758696,-97.476397=0.006295,0.006295=0=1=h=17=A
> >
>
>
> P Please consider the environment before printing this email.
>
> This e-mail and its contents (to include attachments) are the property of
> National Health Systems, Inc., its subsidiaries and affiliates, including
> but not limited to Rx.com Community Healthcare Network, Inc. and its
> subsidiaries, and may contain confidential and proprietary or privileged
> information. If you are not the intended recipient of this e-mail, you are
> hereby notified that any unauthorized disclosure, copying, or distribution
> of this e-mail or of its attachments, or the taking of any unauthorized
> action based on information contained herein is strictly prohibited.
> Unauthorized use of information contained herein may subject you to civil
> and criminal prosecution and penalties. If you are not the intended
> recipient, please immediately notify the sender by telephone at
> 800-433-5719 or return e-mail and permanently delete the original e-mail.
>

Re: offset topics growing huge

[Ali Akhtar]

Newbie question, but what exactly does log.cleaner.enable=true do, and how
do I know if I need to set it to be true?

Also, if config changes like that need to be made once a cluster is up and
running, what's the recommended way to do that? Do you killall -12 kafka
and then make the change, and then start kafka again, one broker at a time?

On Mon, Oct 3, 2016 at 9:27 PM, Tom Crayford  wrote:

> Yes, offset topic compaction is just the normal compaction.
>
> Thanks
>
> Tom Crayford
> Heroku Kafka
>
> On Monday, 3 October 2016, Tobias Adamson  wrote:
>
> > Hi
> > We are using Kafka 0.10.1 with offsets commits being stored inside of
> Kafka
> > After a while these topics become extremely large and we are wondering if
> > we
> > need to enable log.cleaner.enable=true (currently false) to make sure the
> > internal
> > offset topics get compacted and keep their size down?
> >
> > Regards
> > T
>

Re: offset topics growing huge

[Tom Crayford]

Yes, offset topic compaction is just the normal compaction.

Thanks

Tom Crayford
Heroku Kafka

On Monday, 3 October 2016, Tobias Adamson  wrote:

> Hi
> We are using Kafka 0.10.1 with offsets commits being stored inside of Kafka
> After a while these topics become extremely large and we are wondering if
> we
> need to enable log.cleaner.enable=true (currently false) to make sure the
> internal
> offset topics get compacted and keep their size down?
>
> Regards
> T

offset topics growing huge

[Tobias Adamson]

Hi
We are using Kafka 0.10.1 with offsets commits being stored inside of Kafka
After a while these topics become extremely large and we are wondering if we 
need to enable log.cleaner.enable=true (currently false) to make sure the 
internal
offset topics get compacted and keep their size down?

Regards
T

Re: Kafka 0.10.1 ProcessorTopologyTestDriver and WindowedStreamPartitioner issue

[Guozhang Wang]

I have pushed a hotfix to both trunk and 0.10.1, could you check if the
issue is resolved by now?

On Mon, Oct 3, 2016 at 7:18 AM, Hamidreza Afzali <
hamidreza.afz...@hivestreaming.com> wrote:

> Thanks Guozhang. We use ProcessorTopologyTestDriver for unit tests.
>
> Hamid
>
>
> > On 28 Sep 2016, at 11:48 AM, Hamidreza Afzali  hivestreaming.com> wrote:
> >
> > Hi,
> >
> > We are using the latest Kafka 0.10.1 branch. The combination of
> ProcessorTopologyTestDriver and WindowedStreamPartitioner is resulting in a
> division by 0 exception because of the empty list of partitions:
> >
> > https://github.com/apache/kafka/blob/0.10.1/streams/src/
> test/java/org/apache/kafka/test/ProcessorTopologyTestDriver.java#L158
> > https://github.com/apache/kafka/blob/0.10.1/streams/src/
> main/java/org/apache/kafka/streams/kstream/internals/
> WindowedStreamPartitioner.java#L47
> >
> > Our topology looks similar to this:
> >
> >  builder.stream("events")
> >.groupByKey(...)
> >.aggregate(...,
> >  TimeWindows.of(1 * 60 * 1000L)
> >)
> >.mapValues(_.size: Integer)
> >.to(windowedSerde, Serdes.Integer(), "events-over-time")
> >
> > If we use our own partitioner in .to() it works.
> >
> >  class MyStreamPartitioner[K, V]() extends StreamPartitioner[K, V] {
> >override def partition(k: K, v: V, numPartitions: Int): Integer = {
> >  // return an integer between 0 and numPartitions-1, or null if the
> default partitioning logic should be used
> >  null
> >}
> >  }
> >
> > Is this a bug?
> >
> > Thank you in advance,
> > Hamid
> >
>
>


-- 
-- Guozhang

SSL Kafka 0.9.0.1

[Shrikant Patel]

We are are 0.9.0.1 and want to use SSL for ACL and securing communication 
between borker, producer and consumer.

Was \ Is the SSL based ACL in beta for this version of Kafka???


We don't want upgrade to 0.10.x unless it absolutely needed.

Thanks,
Shri
__
Shrikant Patel   |   PDX-NHIN
Enterprise Architecture Team
Asserting the Role of Pharmacy in Healthcare  
www.pdxinc.com
main 817.246.6760 | ext 4302
101 Jim Wright Freeway South, Suite 200, Fort Worth, Texas 
76108-2202


P Please consider the environment before printing this email.

This e-mail and its contents (to include attachments) are the property of 
National Health Systems, Inc., its subsidiaries and affiliates, including but 
not limited to Rx.com Community Healthcare Network, Inc. and its subsidiaries, 
and may contain confidential and proprietary or privileged information. If you 
are not the intended recipient of this e-mail, you are hereby notified that any 
unauthorized disclosure, copying, or distribution of this e-mail or of its 
attachments, or the taking of any unauthorized action based on information 
contained herein is strictly prohibited. Unauthorized use of information 
contained herein may subject you to civil and criminal prosecution and 
penalties. If you are not the intended recipient, please immediately notify the 
sender by telephone at 800-433-5719 or return e-mail and permanently delete the 
original e-mail.

Re: Kafka streaming and topic filter whitelist

[Damian Guy]

They will be automatically added and removed.

On Mon, 3 Oct 2016 at 14:59 Gary Ogden  wrote:

> What if topics are created or deleted after the application has started?
> Will they be added/removed automatically, or do we need to restart the
> application to pick up the changes?
>
> On 1 October 2016 at 04:42, Damian Guy  wrote:
>
> > That is correct.
> >
> > On Fri, 30 Sep 2016 at 18:00 Gary Ogden  wrote:
> >
> > > So how exactly would that work? For example, I can currently do this:
> > >
> > > KStream
> > > textLines = builder.stream(stringSerde, stringSerde,
> > SYSTEM_TOPIC);
> > >
> > > Are you saying that I could put a regex in place of the SYSTEM_TOPIC
> and
> > > that one KStream would be streaming from multiple topics that match
> that
> > > regex?
> > >
> > > If so, that could be useful.
> > >
> > > Gary
> > >
> > >
> > > On 30 September 2016 at 13:35, Damian Guy 
> wrote:
> > >
> > > > Hi Gary,
> > > >
> > > > In the upcoming 0.10.1 release you can do regex subscription - will
> > that
> > > > help?
> > > >
> > > > Thanks,
> > > > Damian
> > > >
> > > > On Fri, 30 Sep 2016 at 14:57 Gary Ogden  wrote:
> > > >
> > > > > Is it possible to use the topic filter whitelist within a Kafka
> > > Streaming
> > > > > application? Or can it only be done in a consumer job?
> > > > >
> > > >
> > >
> >
>

Re: Kafka 0.10.1 ProcessorTopologyTestDriver and WindowedStreamPartitioner issue

[Hamidreza Afzali]

Thanks Guozhang. We use ProcessorTopologyTestDriver for unit tests.

Hamid


> On 28 Sep 2016, at 11:48 AM, Hamidreza Afzali 
>  wrote:
> 
> Hi,
> 
> We are using the latest Kafka 0.10.1 branch. The combination of 
> ProcessorTopologyTestDriver and WindowedStreamPartitioner is resulting in a 
> division by 0 exception because of the empty list of partitions:
> 
> https://github.com/apache/kafka/blob/0.10.1/streams/src/test/java/org/apache/kafka/test/ProcessorTopologyTestDriver.java#L158
> https://github.com/apache/kafka/blob/0.10.1/streams/src/main/java/org/apache/kafka/streams/kstream/internals/WindowedStreamPartitioner.java#L47
> 
> Our topology looks similar to this:
> 
>  builder.stream("events")
>.groupByKey(...)
>.aggregate(...,
>  TimeWindows.of(1 * 60 * 1000L)
>)
>.mapValues(_.size: Integer)
>.to(windowedSerde, Serdes.Integer(), "events-over-time")
> 
> If we use our own partitioner in .to() it works.
> 
>  class MyStreamPartitioner[K, V]() extends StreamPartitioner[K, V] {
>override def partition(k: K, v: V, numPartitions: Int): Integer = {
>  // return an integer between 0 and numPartitions-1, or null if the 
> default partitioning logic should be used
>  null
>}
>  }
> 
> Is this a bug?
> 
> Thank you in advance,
> Hamid
> 

Re: Kafka streaming and topic filter whitelist

[Gary Ogden]

What if topics are created or deleted after the application has started?
Will they be added/removed automatically, or do we need to restart the
application to pick up the changes?

On 1 October 2016 at 04:42, Damian Guy  wrote:

> That is correct.
>
> On Fri, 30 Sep 2016 at 18:00 Gary Ogden  wrote:
>
> > So how exactly would that work? For example, I can currently do this:
> >
> > KStream
> > textLines = builder.stream(stringSerde, stringSerde,
> SYSTEM_TOPIC);
> >
> > Are you saying that I could put a regex in place of the SYSTEM_TOPIC and
> > that one KStream would be streaming from multiple topics that match that
> > regex?
> >
> > If so, that could be useful.
> >
> > Gary
> >
> >
> > On 30 September 2016 at 13:35, Damian Guy  wrote:
> >
> > > Hi Gary,
> > >
> > > In the upcoming 0.10.1 release you can do regex subscription - will
> that
> > > help?
> > >
> > > Thanks,
> > > Damian
> > >
> > > On Fri, 30 Sep 2016 at 14:57 Gary Ogden  wrote:
> > >
> > > > Is it possible to use the topic filter whitelist within a Kafka
> > Streaming
> > > > application? Or can it only be done in a consumer job?
> > > >
> > >
> >
>

kafka streams with dynamic content and filtering

[Gary Ogden]

I have a use case, and I'm wondering if it's possible to do this with Kafka.

Let's say we will have customers that will be uploading JSON to our system,
but that JSON layout will be different between each customer. They are able
to define the schema of the JSON being uploaded.

They will then be able to define the fields in that JSON they want to
gather metrics on (sum, counts etc).

Is there a way with Kafka streaming to dynamically read the configuration
for that customer and process the json and do counts and sums for the
fields they've defined.

It's possible at any time they may want to modify the configuration for
their json as well. Stop counting one field, start counting another.

They will also want to do some inferences as well. IE, if this particular
JSON is uploaded with a field in it, then check to see if another json was
uploaded within 8 hours.

Is it possible for Kafka streaming to be this dynamic?

Re: Is Kafka 8 user compatible with kafka 10 broker?

[Gerard Klijs]

I think, but don't know for sure, it doesn't matter for consumers, since
the messages you read are still 'old' images. I would expect errors van you
use an old producer, and/or when consuming the record from the old producer.

On Mon, Oct 3, 2016 at 7:09 AM Nikhil Goyal  wrote:

> Hi guys,
>
> I created a new cluster with kafka 10 broker. I have the following settings
> in server properties file:
>
> inter.broker.protocol.version=0.10.0.0
> message.format.version=0.10.0
> log.message.format.version=0.10.0
>
> When I try to read from old console consumer it still succeeds. Online
> documents say old consumer is not compatible as long as message format
> version is not set to 0.8.0. So I wanted to ask if I made a mistake
> somewhere in the server_properties or if this is the expected behavior?
>
> Thanks
> Nikhil
>