Re: How to know which consumer member causes rebalance

[Yifan Ying]

Some more details about the consumer setup:

   - version 0.10.1.1
   - the topic has 400 partitions
   - consumers are running on 200 mesos instances, and each instance is
   running 2 KafkaConsumer threads


Yifan

On Wed, Mar 8, 2017 at 11:46 PM, Yifan Ying  wrote:

> Hi,
>
> We are running a group of Kafka consumers on 200 mesos instances and we
> are observing constant Revoking/Rejoining in our consumer logs. But it's
> hard to tell which consumer member initially caused this issue as every
> consumer needs re-join in this case. Is there a good way to find out that
> 'bad' consumer?
>
> Thanks.
>
> --
> Yifan
>
>
>


-- 
Yifan

How to know which consumer member causes rebalance

[Yifan Ying]

Hi,

We are running a group of Kafka consumers on 200 mesos instances and we are
observing constant Revoking/Rejoining in our consumer logs. But it's hard
to tell which consumer member initially caused this issue as every consumer
needs re-join in this case. Is there a good way to find out that 'bad'
consumer?

Thanks.

-- 
Yifan

Re: How to set offset for a consumer in Kafka 0.10.0.X

[Sudev A C]

Hi Glen,

Maybe have a look at this, slightly different version. Not doing anything
more than what you have seen in gist.

You should pass broker url as first parameter and and a json containing
desired offsets as second parameter.

https://github.com/goibibo/woof/tree/master/scripts/offset_commit_manual

As suggested in previous mail it will be good to shut all consumers before
reseting the offset.

Thanks
Sudev
On Thu, 9 Mar 2017 at 11:41 AM, Manikumar  wrote:

> A tool for reseting consumer group offsets is proposed here:
> KIP-122:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling
>
> KIP is currently in the voting stage.
>
> On Thu, Mar 9, 2017 at 7:35 AM, Jeff Widman  wrote:
>
> > Did you have to do anything different beyond what was already in the
> Gist?
> >
> > I'd be curious to see the code as I've considered putting together a
> small
> > repo of various python scripts I've found useful when working with Kafka
> >
> > On Wed, Mar 8, 2017 at 1:11 PM, Glen Ogilvie 
> > wrote:
> >
> > > Thank you Jeff and Robert.
> > >
> > > I've had success in getting the offset position to seek to a position
> in
> > > the queue, when all other consumers in the group are off.
> > >
> > > Would anyone like me to tidy the tool up enough that it could be
> included
> > > in Kafka or the docs?
> > >
> > > Regards
> > > --
> > > Glen Ogilvie
> > > Open Systems Specialists
> > > Level 1, 162 Grafton Road
> > > http://www.oss.co.nz/
> > >
> > > Ph: +64 9 984 3000
> > > Mobile: +64 21 684 146
> > > GPG Key: ACED9C17
> > >
> > >
> > > From: "Robert Quinlivan" 
> > > To: "users" 
> > > Sent: Thursday, 9 March, 2017 6:42:15 AM
> > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> > >
> > > The best approach would be:
> > > - Have all consumers in your group shut down
> > > - Have an offset reset tool join with the same group name as above
> > > - Offset tool subscribes to all topic-partitions, seeks to the desired
> > > offset, and commits.
> > > - Offset tool shuts down
> > > - Consumers then restart and re-join the consumer group, resuming at
> the
> > > offsets that were last committed for each topic-partition
> > >
> > > On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman 
> wrote:
> > >
> > > > Yeah, that gist looks like it *should* work. I haven't tested it so
> > can't
> > > > guarantee.
> > > >
> > > > On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie  >
> > > > wrote:
> > > >
> > > > > Hi Jeff,
> > > > >
> > > > > Yes, the work I am doing is ops work. Logstash is consuming from
> the
> > > > topic
> > > > > + consumer group, and I don't want it to start at the beginning,
> but
> > > > rather
> > > > > at a specific offset,
> > > > > so setting the offset for the consumer group externally, then
> > starting
> > > up
> > > > > logstash is my goal.
> > > > >
> > > > > I'm still a little unclear as to how to do this.
> > > > >
> > > > > Is this python script: http://pastebin.com/tvxj1wTX
> > > > >
> > > > > The right way to go about getting the offset set to a specific
> value
> > > > > (12345678 in this example) for a specific consumer group?
> > > > >
> > > > > Regards
> > > > > --
> > > > > Glen Ogilvie
> > > > > Open Systems Specialists
> > > > > Level 1, 162 Grafton Road
> > > > > http://www.oss.co.nz/
> > > > >
> > > > > Ph: +64 9 984 3000
> > > > > Mobile: +64 21 684 146
> > > > > GPG Key: ACED9C17
> > > > >
> > > > >
> > > > > From: "Jeff Widman" 
> > > > > To: "users" 
> > > > > Sent: Wednesday, 8 March, 2017 1:41:17 PM
> > > > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> > > > >
> > > > > Offsets for modern kafka consumers are stored in an internal Kafka
> > > topic,
> > > > > so they aren't as easy to change as zookeeper.
> > > > >
> > > > > To set a consumer offset, you need a consumer within a consumer
> group
> > > to
> > > > > call commit() with your explicit offset. If needed, you can create
> a
> > > > dummy
> > > > > consumer and tell it to join an existing consumer group.
> > > > >
> > > > > Take a look at this migration script, especially the part where it
> > > > commits
> > > > > the offset to see how it can work in Scala:
> > > > > https://github.com/apache/kafka/pull/2615/files
> > > > >
> > > > > It's fairly straightforward to do this within most other clients as
> > > well.
> > > > > If you're doing some quick ops work where you don't want to spin up
> > the
> > > > > JVM, then it's fairly easy to do this using kafka-python.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <
> glen.ogil...@oss.co.nz
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to
> figure
> > > out
> > > > > if
> > > > > > 

Re: How to set offset for a consumer in Kafka 0.10.0.X

[Manikumar]

A tool for reseting consumer group offsets is proposed here:
KIP-122:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling

KIP is currently in the voting stage.

On Thu, Mar 9, 2017 at 7:35 AM, Jeff Widman  wrote:

> Did you have to do anything different beyond what was already in the Gist?
>
> I'd be curious to see the code as I've considered putting together a small
> repo of various python scripts I've found useful when working with Kafka
>
> On Wed, Mar 8, 2017 at 1:11 PM, Glen Ogilvie 
> wrote:
>
> > Thank you Jeff and Robert.
> >
> > I've had success in getting the offset position to seek to a position in
> > the queue, when all other consumers in the group are off.
> >
> > Would anyone like me to tidy the tool up enough that it could be included
> > in Kafka or the docs?
> >
> > Regards
> > --
> > Glen Ogilvie
> > Open Systems Specialists
> > Level 1, 162 Grafton Road
> > http://www.oss.co.nz/
> >
> > Ph: +64 9 984 3000
> > Mobile: +64 21 684 146
> > GPG Key: ACED9C17
> >
> >
> > From: "Robert Quinlivan" 
> > To: "users" 
> > Sent: Thursday, 9 March, 2017 6:42:15 AM
> > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> >
> > The best approach would be:
> > - Have all consumers in your group shut down
> > - Have an offset reset tool join with the same group name as above
> > - Offset tool subscribes to all topic-partitions, seeks to the desired
> > offset, and commits.
> > - Offset tool shuts down
> > - Consumers then restart and re-join the consumer group, resuming at the
> > offsets that were last committed for each topic-partition
> >
> > On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman  wrote:
> >
> > > Yeah, that gist looks like it *should* work. I haven't tested it so
> can't
> > > guarantee.
> > >
> > > On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie 
> > > wrote:
> > >
> > > > Hi Jeff,
> > > >
> > > > Yes, the work I am doing is ops work. Logstash is consuming from the
> > > topic
> > > > + consumer group, and I don't want it to start at the beginning, but
> > > rather
> > > > at a specific offset,
> > > > so setting the offset for the consumer group externally, then
> starting
> > up
> > > > logstash is my goal.
> > > >
> > > > I'm still a little unclear as to how to do this.
> > > >
> > > > Is this python script: http://pastebin.com/tvxj1wTX
> > > >
> > > > The right way to go about getting the offset set to a specific value
> > > > (12345678 in this example) for a specific consumer group?
> > > >
> > > > Regards
> > > > --
> > > > Glen Ogilvie
> > > > Open Systems Specialists
> > > > Level 1, 162 Grafton Road
> > > > http://www.oss.co.nz/
> > > >
> > > > Ph: +64 9 984 3000
> > > > Mobile: +64 21 684 146
> > > > GPG Key: ACED9C17
> > > >
> > > >
> > > > From: "Jeff Widman" 
> > > > To: "users" 
> > > > Sent: Wednesday, 8 March, 2017 1:41:17 PM
> > > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> > > >
> > > > Offsets for modern kafka consumers are stored in an internal Kafka
> > topic,
> > > > so they aren't as easy to change as zookeeper.
> > > >
> > > > To set a consumer offset, you need a consumer within a consumer group
> > to
> > > > call commit() with your explicit offset. If needed, you can create a
> > > dummy
> > > > consumer and tell it to join an existing consumer group.
> > > >
> > > > Take a look at this migration script, especially the part where it
> > > commits
> > > > the offset to see how it can work in Scala:
> > > > https://github.com/apache/kafka/pull/2615/files
> > > >
> > > > It's fairly straightforward to do this within most other clients as
> > well.
> > > > If you're doing some quick ops work where you don't want to spin up
> the
> > > > JVM, then it's fairly easy to do this using kafka-python.
> > > >
> > > >
> > > >
> > > > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie  >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure
> > out
> > > > if
> > > > > I can manually
> > > > > set a consumer offset, for a specific consumer when that consumer
> is
> > > > > stopped.
> > > > >
> > > > > It looks like it used to be done using: kafka.tools.ExportZkOffsets
> > and
> > > > > kafka.tools.ImportZkOffsets
> > > > > ( https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > System+Tools#SystemTools-ConsumerOffsetChecker )
> > > > >
> > > > > However, if my version they don't work, because they try and read
> > from
> > > > > zookeeper /consumers which is empty.. I think they are old tools.
> > > > >
> > > > > Does anyone know where in zookeeper, where the current kafka keeps
> > > > > consumer offsets?
> > > > >
> > > > > Regards
> > > > > --
> > > > > Glen Ogilvie
> > > > > Open Systems Specialists
> > > > > Level 1, 162 

performance regression with 0.10.2.0 client talking to 0.10.0.1 servers

[Alex Cozzi]

I noticed a significant reduction in throughput when I upgraded my app to 
0.10.2.0. I am talking to 0.10.0.1 server and I wanted to take advantage of the 
compatibility of the newest release with old server, but I noticed that my 
performance went from 20K msg/sec to 2K msg/sec in one case and from 9K msg/sec 
to 4K msg/sec in another case.
Is this expected or should I file a bug?
Alex

Re: Settings at cluster level: retention and cleanup policy

[Zakee]

Hi Nico,

They can be defined at both cluster and topics levels. Refer 
https://kafka.apache.org/documentation/#topic-config for topic-level overrides 
available.

Cheers! 
-Z

> On Mar 8, 2017, at 12:41 PM, Nicolas MOTTE  wrote:
> 
> Hi everyone,
> 
> Is there any reason why retention and cleanup policy are defined at cluster 
> level and not topic level?
> I can t see why it would not be possible from a technical point of view...
> 
> Cheers
> Nico
> 


Brilliant Trick Melts Body Fat Overnight (Do This Tonight)
Biblical Belly Breakthrough
http://thirdpartyoffers.netzero.net/TGL3231/58c0c068c9d38406811e4st02vuc

Re: How to set offset for a consumer in Kafka 0.10.0.X

[Jeff Widman]

Did you have to do anything different beyond what was already in the Gist?

I'd be curious to see the code as I've considered putting together a small
repo of various python scripts I've found useful when working with Kafka

On Wed, Mar 8, 2017 at 1:11 PM, Glen Ogilvie  wrote:

> Thank you Jeff and Robert.
>
> I've had success in getting the offset position to seek to a position in
> the queue, when all other consumers in the group are off.
>
> Would anyone like me to tidy the tool up enough that it could be included
> in Kafka or the docs?
>
> Regards
> --
> Glen Ogilvie
> Open Systems Specialists
> Level 1, 162 Grafton Road
> http://www.oss.co.nz/
>
> Ph: +64 9 984 3000
> Mobile: +64 21 684 146
> GPG Key: ACED9C17
>
>
> From: "Robert Quinlivan" 
> To: "users" 
> Sent: Thursday, 9 March, 2017 6:42:15 AM
> Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
>
> The best approach would be:
> - Have all consumers in your group shut down
> - Have an offset reset tool join with the same group name as above
> - Offset tool subscribes to all topic-partitions, seeks to the desired
> offset, and commits.
> - Offset tool shuts down
> - Consumers then restart and re-join the consumer group, resuming at the
> offsets that were last committed for each topic-partition
>
> On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman  wrote:
>
> > Yeah, that gist looks like it *should* work. I haven't tested it so can't
> > guarantee.
> >
> > On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie 
> > wrote:
> >
> > > Hi Jeff,
> > >
> > > Yes, the work I am doing is ops work. Logstash is consuming from the
> > topic
> > > + consumer group, and I don't want it to start at the beginning, but
> > rather
> > > at a specific offset,
> > > so setting the offset for the consumer group externally, then starting
> up
> > > logstash is my goal.
> > >
> > > I'm still a little unclear as to how to do this.
> > >
> > > Is this python script: http://pastebin.com/tvxj1wTX
> > >
> > > The right way to go about getting the offset set to a specific value
> > > (12345678 in this example) for a specific consumer group?
> > >
> > > Regards
> > > --
> > > Glen Ogilvie
> > > Open Systems Specialists
> > > Level 1, 162 Grafton Road
> > > http://www.oss.co.nz/
> > >
> > > Ph: +64 9 984 3000
> > > Mobile: +64 21 684 146
> > > GPG Key: ACED9C17
> > >
> > >
> > > From: "Jeff Widman" 
> > > To: "users" 
> > > Sent: Wednesday, 8 March, 2017 1:41:17 PM
> > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> > >
> > > Offsets for modern kafka consumers are stored in an internal Kafka
> topic,
> > > so they aren't as easy to change as zookeeper.
> > >
> > > To set a consumer offset, you need a consumer within a consumer group
> to
> > > call commit() with your explicit offset. If needed, you can create a
> > dummy
> > > consumer and tell it to join an existing consumer group.
> > >
> > > Take a look at this migration script, especially the part where it
> > commits
> > > the offset to see how it can work in Scala:
> > > https://github.com/apache/kafka/pull/2615/files
> > >
> > > It's fairly straightforward to do this within most other clients as
> well.
> > > If you're doing some quick ops work where you don't want to spin up the
> > > JVM, then it's fairly easy to do this using kafka-python.
> > >
> > >
> > >
> > > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure
> out
> > > if
> > > > I can manually
> > > > set a consumer offset, for a specific consumer when that consumer is
> > > > stopped.
> > > >
> > > > It looks like it used to be done using: kafka.tools.ExportZkOffsets
> and
> > > > kafka.tools.ImportZkOffsets
> > > > ( https://cwiki.apache.org/confluence/display/KAFKA/
> > > > System+Tools#SystemTools-ConsumerOffsetChecker )
> > > >
> > > > However, if my version they don't work, because they try and read
> from
> > > > zookeeper /consumers which is empty.. I think they are old tools.
> > > >
> > > > Does anyone know where in zookeeper, where the current kafka keeps
> > > > consumer offsets?
> > > >
> > > > Regards
> > > > --
> > > > Glen Ogilvie
> > > > Open Systems Specialists
> > > > Level 1, 162 Grafton Road
> > > > http://www.oss.co.nz/
> > > >
> > > > Ph: +64 9 984 3000
> > > > Mobile: +64 21 684 146
> > > > GPG Key: ACED9C17
> > > >
> > >
> >
>
>
>
> --
> Robert Quinlivan
> Software Engineer, Signal
>

Re: Performance and encryption

[Stephane Maarek]

I believe these are defaults you can set at the broker level so that if the
topic doesn’t have that setting set, it will inherit those
But you can definitely override your topic configuration at the topic level

On 9 March 2017 at 7:42:14 am, Nicolas Motte (lingusi...@gmail.com) wrote:

Hi everyone, I have another question.
Is there any reason why retention and cleanup policy are defined at cluster
level and not topic level?
I can t see why it would not be possible from a technical point of view...

2017-03-06 14:38 GMT+01:00 Nicolas Motte :

> Hi everyone,
>
> I understand one of the reasons why Kafka is performant is by using
> zero-copy.
>
> I often hear that when encryption is enabled, then Kafka has to copy the
> data in user space to decode the message, so it has a big impact on
> performance.
>
> If it is true, I don t get why the message has to be decoded by Kafka. I
> would assume that whether the message is encrypted or not, Kafka simply
> receives it, appends it to the file, and when a consumer wants to read
it,
> it simply reads at the right offset...
>
> Also I m wondering if it s the case if we don t use keys (pure queuing
> system with key=null).
>
> Cheers
> Nico
>
>

Re: Performance and encryption

[Todd Palino]

They are defined at the broker level as a default for all topics that do
not have an override for those configs. Both (and many other configs) can
be overridden for individual topics using the command line tools.

-Todd


On Wed, Mar 8, 2017 at 12:36 PM, Nicolas Motte  wrote:

> Hi everyone, I have another question.
> Is there any reason why retention and cleanup policy are defined at cluster
> level and not topic level?
> I can t see why it would not be possible from a technical point of view...
>
> 2017-03-06 14:38 GMT+01:00 Nicolas Motte :
>
> > Hi everyone,
> >
> > I understand one of the reasons why Kafka is performant is by using
> > zero-copy.
> >
> > I often hear that when encryption is enabled, then Kafka has to copy the
> > data in user space to decode the message, so it has a big impact on
> > performance.
> >
> > If it is true, I don t get why the message has to be decoded by Kafka. I
> > would assume that whether the message is encrypted or not, Kafka simply
> > receives it, appends it to the file, and when a consumer wants to read
> it,
> > it simply reads at the right offset...
> >
> > Also I m wondering if it s the case if we don t use keys (pure queuing
> > system with key=null).
> >
> > Cheers
> > Nico
> >
> >
>



-- 
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming



linkedin.com/in/toddpalino

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

[Matthias J. Sax]

Hi,

sorry for not replying earlier and thanks for all your feedback. After
some more discussions I updated the KIP. The new proposal puts some
other design considerations into account, that I want to highlight
shortly. Those considerations, automatically resolve the concerns raised.

First some answers:

> The PAPI processors I use in my KStreams app are all functioning on KTable
> internals.  I wouldn't be able to convert them to process()/transform().
> 
> What's the harm in permitting both APIs to be used in the same application?

It's not about "harm" but about design. We want to switch from a
"inheritance" to a "composition" pattern.

About the interface idea: using a shared interface would not help to get
a composition pattern


Next I want to give the design considerations leading to the updated KIP:

1) Using KStreamBuilder in the constructor of KafkaStreams is unnatural.
KafkaStreams client executes a `Topology` and this execution should be
independent of the way the topology is "put together", ie, low-level API
or DSL.

2) Thus, we don't want to have any changes to KafkaStreams class.

3) Thus, KStreamBuilder needs to have a method `build()` that returns a
`Topology` that can be passed into KafakStreams.

4) Because `KStreamBuilder` should build a `Topology` I suggest to
rename the new class to `StreamsTopologyBuilder` (the name
TopologyBuilder would actually be more natural, but would be easily
confused with old low-level API TopologyBuilder).

Thus, PAPI and DSL can be mixed-and-matched with full power, as
StreamsTopologyBuilder return the created Topology via #build().

I also removed `final` for both builder classes.



With regard to the larger scope of the overal API redesign, I also want
to point to a summary of API issues:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Discussions

Thus, this KIP is only one building block of a larger improvement
effort, and we hope to get as much as possible done for 0.11. If you
have any API improvement ideas, please share them so we can come up with
an holistic sound design (instead of uncoordinated local improvements
that might diverge)



Looking forward to your feedback on this KIP and the other API issues.



-Matthias




On 2/15/17 7:36 PM, Mathieu Fenniak wrote:
> On Wed, Feb 15, 2017 at 5:04 PM, Matthias J. Sax 
> wrote:
> 
>> - We also removed method #topologyBuilder() from KStreamBuilder because
>> we think #transform() should provide all functionality you need to
>> mix-an-match Processor API and DSL. If there is any further concern
>> about this, please let us know.
>>
> 
> Hi Matthias,
> 
> Yes, I'm sorry I didn't respond sooner, but I still have a lot of concerns
> about this.  You're correct to point out that transform() can be used for
> some of the output situations I pointed out; albeit it seems somewhat
> awkward to do so in a "transform" method; what do you do with the retval?
> 
> The PAPI processors I use in my KStreams app are all functioning on KTable
> internals.  I wouldn't be able to convert them to process()/transform().
> 
> What's the harm in permitting both APIs to be used in the same application?
> 
> Mathieu
> 



signature.asc
Description: OpenPGP digital signature

Re: Performance and Encryption

[Todd Palino]

Nicholas, this appears to be a duplicate of your question from 2 days ago.
Please review that for discussion on this question.

-Todd


On Wed, Mar 8, 2017 at 1:08 PM, Tauzell, Dave 
wrote:

> I think because the product batches messages which could be for different
> topics.
>
> -Dave
>
> -Original Message-
> From: Nicolas MOTTE [mailto:nicolas.mo...@amadeus.com]
> Sent: Wednesday, March 8, 2017 2:41 PM
> To: users@kafka.apache.org
> Subject: Performance and Encryption
>
> Hi everyone,
>
> I understand one of the reasons why Kafka is performant is by using
> zero-copy.
>
> I often hear that when encryption is enabled, then Kafka has to copy the
> data in user space to decode the message, so it has a big impact on
> performance.
>
> If it is true, I don t get why the message has to be decoded by Kafka. I
> would assume that whether the message is encrypted or not, Kafka simply
> receives it, appends it to the file, and when a consumer wants to read it,
> it simply reads at the right offset...
>
> Also I m wondering if it s the case if we don t use keys (pure queuing
> system with key=null).
>
> Cheers
> Nico
>
> This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of the
> individual or entity to whom they are addressed. If you have received this
> e-mail in error, please notify the sender by reply e-mail immediately and
> destroy all copies of the e-mail and any attachments.
>
>


-- 
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming



linkedin.com/in/toddpalino

Re: Settings at cluster level: retention and cleanup policy

[Kaufman Ng]

Actually they can be at topic level configs as well. If they are not
defined then the server level configs are used. From Apache Kafka doc:

"The server's default configuration for this property is given under the
Server Default Property heading. A given server default config value only
applies to a topic if it does not have an explicit topic config override."

On Wed, Mar 8, 2017 at 3:41 PM, Nicolas MOTTE 
wrote:

> Hi everyone,
>
> Is there any reason why retention and cleanup policy are defined at
> cluster level and not topic level?
> I can t see why it would not be possible from a technical point of view...
>
> Cheers
> Nico
>
>


-- 
Kaufman Ng
+1 646 961 8063
Solutions Architect | Confluent | www.confluent.io

Re: How to set offset for a consumer in Kafka 0.10.0.X

[Glen Ogilvie]

Thank you Jeff and Robert. 

I've had success in getting the offset position to seek to a position in the 
queue, when all other consumers in the group are off. 

Would anyone like me to tidy the tool up enough that it could be included in 
Kafka or the docs? 

Regards 
-- 
Glen Ogilvie 
Open Systems Specialists 
Level 1, 162 Grafton Road 
http://www.oss.co.nz/ 

Ph: +64 9 984 3000 
Mobile: +64 21 684 146 
GPG Key: ACED9C17 


From: "Robert Quinlivan"  
To: "users"  
Sent: Thursday, 9 March, 2017 6:42:15 AM 
Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X 

The best approach would be: 
- Have all consumers in your group shut down 
- Have an offset reset tool join with the same group name as above 
- Offset tool subscribes to all topic-partitions, seeks to the desired 
offset, and commits. 
- Offset tool shuts down 
- Consumers then restart and re-join the consumer group, resuming at the 
offsets that were last committed for each topic-partition 

On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman  wrote: 

> Yeah, that gist looks like it *should* work. I haven't tested it so can't 
> guarantee. 
> 
> On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie  
> wrote: 
> 
> > Hi Jeff, 
> > 
> > Yes, the work I am doing is ops work. Logstash is consuming from the 
> topic 
> > + consumer group, and I don't want it to start at the beginning, but 
> rather 
> > at a specific offset, 
> > so setting the offset for the consumer group externally, then starting up 
> > logstash is my goal. 
> > 
> > I'm still a little unclear as to how to do this. 
> > 
> > Is this python script: http://pastebin.com/tvxj1wTX 
> > 
> > The right way to go about getting the offset set to a specific value 
> > (12345678 in this example) for a specific consumer group? 
> > 
> > Regards 
> > -- 
> > Glen Ogilvie 
> > Open Systems Specialists 
> > Level 1, 162 Grafton Road 
> > http://www.oss.co.nz/ 
> > 
> > Ph: +64 9 984 3000 
> > Mobile: +64 21 684 146 
> > GPG Key: ACED9C17 
> > 
> > 
> > From: "Jeff Widman"  
> > To: "users"  
> > Sent: Wednesday, 8 March, 2017 1:41:17 PM 
> > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X 
> > 
> > Offsets for modern kafka consumers are stored in an internal Kafka topic, 
> > so they aren't as easy to change as zookeeper. 
> > 
> > To set a consumer offset, you need a consumer within a consumer group to 
> > call commit() with your explicit offset. If needed, you can create a 
> dummy 
> > consumer and tell it to join an existing consumer group. 
> > 
> > Take a look at this migration script, especially the part where it 
> commits 
> > the offset to see how it can work in Scala: 
> > https://github.com/apache/kafka/pull/2615/files 
> > 
> > It's fairly straightforward to do this within most other clients as well. 
> > If you're doing some quick ops work where you don't want to spin up the 
> > JVM, then it's fairly easy to do this using kafka-python. 
> > 
> > 
> > 
> > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie  
> > wrote: 
> > 
> > > Hi, 
> > > 
> > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure out 
> > if 
> > > I can manually 
> > > set a consumer offset, for a specific consumer when that consumer is 
> > > stopped. 
> > > 
> > > It looks like it used to be done using: kafka.tools.ExportZkOffsets and 
> > > kafka.tools.ImportZkOffsets 
> > > ( https://cwiki.apache.org/confluence/display/KAFKA/ 
> > > System+Tools#SystemTools-ConsumerOffsetChecker ) 
> > > 
> > > However, if my version they don't work, because they try and read from 
> > > zookeeper /consumers which is empty.. I think they are old tools. 
> > > 
> > > Does anyone know where in zookeeper, where the current kafka keeps 
> > > consumer offsets? 
> > > 
> > > Regards 
> > > -- 
> > > Glen Ogilvie 
> > > Open Systems Specialists 
> > > Level 1, 162 Grafton Road 
> > > http://www.oss.co.nz/ 
> > > 
> > > Ph: +64 9 984 3000 
> > > Mobile: +64 21 684 146 
> > > GPG Key: ACED9C17 
> > > 
> > 
> 



-- 
Robert Quinlivan 
Software Engineer, Signal 

Re: Performance and encryption

[Nicolas Motte]

Hi everyone, I have another question.
Is there any reason why retention and cleanup policy are defined at cluster
level and not topic level?
I can t see why it would not be possible from a technical point of view...

2017-03-06 14:38 GMT+01:00 Nicolas Motte :

> Hi everyone,
>
> I understand one of the reasons why Kafka is performant is by using
> zero-copy.
>
> I often hear that when encryption is enabled, then Kafka has to copy the
> data in user space to decode the message, so it has a big impact on
> performance.
>
> If it is true, I don t get why the message has to be decoded by Kafka. I
> would assume that whether the message is encrypted or not, Kafka simply
> receives it, appends it to the file, and when a consumer wants to read it,
> it simply reads at the right offset...
>
> Also I m wondering if it s the case if we don t use keys (pure queuing
> system with key=null).
>
> Cheers
> Nico
>
>

Settings at cluster level: retention and cleanup policy

[Nicolas MOTTE]

Hi everyone,

Is there any reason why retention and cleanup policy are defined at cluster 
level and not topic level?
I can t see why it would not be possible from a technical point of view...

Cheers
Nico

Performance and Encryption

[Nicolas MOTTE]

Hi everyone,

I understand one of the reasons why Kafka is performant is by using zero-copy.

I often hear that when encryption is enabled, then Kafka has to copy the data 
in user space to decode the message, so it has a big impact on performance.

If it is true, I don t get why the message has to be decoded by Kafka. I would 
assume that whether the message is encrypted or not, Kafka simply receives it, 
appends it to the file, and when a consumer wants to read it, it simply reads 
at the right offset...

Also I m wondering if it s the case if we don t use keys (pure queuing system 
with key=null).

Cheers
Nico

Configuring an IKEv2 endpoint with PAM authentication

[Prez Cannady]

I'm currently stuck trying to configure strongswan to support road warriors and 
for workstations operated by other remote users.  I believe I accomplish this 
with IKEv2, EAP and xauth-pam.

To that end:

1. I've setup an Ubuntu 16.04 box, 
2. installed strongswan, strongswan-plugin-xauth, strong swan-plugin-xauth-pam, 
and strongswan-plugin-xauth-eap, and
3. generated the key and X.509 certificate for the endpoint

I've yet to touch /etc/ipsec.conf, /etc/ipsec.secrets or /etc/strongswan.conf.  

Right now, I'm not sure what to do next. I'd appreciate any pointers to example 
configurations that may come close to try I'm achieve, or some indication that 
I'm going down a rabbit hole.



Prez Cannady  
e: revp...@opencorrelate.org   
h: https://revprez.github.io 

Re: How to set offset for a consumer in Kafka 0.10.0.X

[Robert Quinlivan]

The best approach would be:
- Have all consumers in your group shut down
- Have an offset reset tool join with the same group name as above
- Offset tool subscribes to all topic-partitions, seeks to the desired
offset, and commits.
- Offset tool shuts down
- Consumers then restart and re-join the consumer group, resuming at the
offsets that were last committed for each topic-partition

On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman  wrote:

> Yeah, that gist looks like it *should* work. I haven't tested it so can't
> guarantee.
>
> On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie 
> wrote:
>
> > Hi Jeff,
> >
> > Yes, the work I am doing is ops work. Logstash is consuming from the
> topic
> > + consumer group, and I don't want it to start at the beginning, but
> rather
> > at a specific offset,
> > so setting the offset for the consumer group externally, then starting up
> > logstash is my goal.
> >
> > I'm still a little unclear as to how to do this.
> >
> > Is this python script: http://pastebin.com/tvxj1wTX
> >
> > The right way to go about getting the offset set to a specific value
> > (12345678 in this example) for a specific consumer group?
> >
> > Regards
> > --
> > Glen Ogilvie
> > Open Systems Specialists
> > Level 1, 162 Grafton Road
> > http://www.oss.co.nz/
> >
> > Ph: +64 9 984 3000
> > Mobile: +64 21 684 146
> > GPG Key: ACED9C17
> >
> >
> > From: "Jeff Widman" 
> > To: "users" 
> > Sent: Wednesday, 8 March, 2017 1:41:17 PM
> > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> >
> > Offsets for modern kafka consumers are stored in an internal Kafka topic,
> > so they aren't as easy to change as zookeeper.
> >
> > To set a consumer offset, you need a consumer within a consumer group to
> > call commit() with your explicit offset. If needed, you can create a
> dummy
> > consumer and tell it to join an existing consumer group.
> >
> > Take a look at this migration script, especially the part where it
> commits
> > the offset to see how it can work in Scala:
> > https://github.com/apache/kafka/pull/2615/files
> >
> > It's fairly straightforward to do this within most other clients as well.
> > If you're doing some quick ops work where you don't want to spin up the
> > JVM, then it's fairly easy to do this using kafka-python.
> >
> >
> >
> > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie 
> > wrote:
> >
> > > Hi,
> > >
> > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure out
> > if
> > > I can manually
> > > set a consumer offset, for a specific consumer when that consumer is
> > > stopped.
> > >
> > > It looks like it used to be done using: kafka.tools.ExportZkOffsets and
> > > kafka.tools.ImportZkOffsets
> > > ( https://cwiki.apache.org/confluence/display/KAFKA/
> > > System+Tools#SystemTools-ConsumerOffsetChecker )
> > >
> > > However, if my version they don't work, because they try and read from
> > > zookeeper /consumers which is empty.. I think they are old tools.
> > >
> > > Does anyone know where in zookeeper, where the current kafka keeps
> > > consumer offsets?
> > >
> > > Regards
> > > --
> > > Glen Ogilvie
> > > Open Systems Specialists
> > > Level 1, 162 Grafton Road
> > > http://www.oss.co.nz/
> > >
> > > Ph: +64 9 984 3000
> > > Mobile: +64 21 684 146
> > > GPG Key: ACED9C17
> > >
> >
>



-- 
Robert Quinlivan
Software Engineer, Signal

Re: How to set offset for a consumer in Kafka 0.10.0.X

[Jeff Widman]

Yeah, that gist looks like it *should* work. I haven't tested it so can't
guarantee.

On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie  wrote:

> Hi Jeff,
>
> Yes, the work I am doing is ops work. Logstash is consuming from the topic
> + consumer group, and I don't want it to start at the beginning, but rather
> at a specific offset,
> so setting the offset for the consumer group externally, then starting up
> logstash is my goal.
>
> I'm still a little unclear as to how to do this.
>
> Is this python script: http://pastebin.com/tvxj1wTX
>
> The right way to go about getting the offset set to a specific value
> (12345678 in this example) for a specific consumer group?
>
> Regards
> --
> Glen Ogilvie
> Open Systems Specialists
> Level 1, 162 Grafton Road
> http://www.oss.co.nz/
>
> Ph: +64 9 984 3000
> Mobile: +64 21 684 146
> GPG Key: ACED9C17
>
>
> From: "Jeff Widman" 
> To: "users" 
> Sent: Wednesday, 8 March, 2017 1:41:17 PM
> Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
>
> Offsets for modern kafka consumers are stored in an internal Kafka topic,
> so they aren't as easy to change as zookeeper.
>
> To set a consumer offset, you need a consumer within a consumer group to
> call commit() with your explicit offset. If needed, you can create a dummy
> consumer and tell it to join an existing consumer group.
>
> Take a look at this migration script, especially the part where it commits
> the offset to see how it can work in Scala:
> https://github.com/apache/kafka/pull/2615/files
>
> It's fairly straightforward to do this within most other clients as well.
> If you're doing some quick ops work where you don't want to spin up the
> JVM, then it's fairly easy to do this using kafka-python.
>
>
>
> On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie 
> wrote:
>
> > Hi,
> >
> > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure out
> if
> > I can manually
> > set a consumer offset, for a specific consumer when that consumer is
> > stopped.
> >
> > It looks like it used to be done using: kafka.tools.ExportZkOffsets and
> > kafka.tools.ImportZkOffsets
> > ( https://cwiki.apache.org/confluence/display/KAFKA/
> > System+Tools#SystemTools-ConsumerOffsetChecker )
> >
> > However, if my version they don't work, because they try and read from
> > zookeeper /consumers which is empty.. I think they are old tools.
> >
> > Does anyone know where in zookeeper, where the current kafka keeps
> > consumer offsets?
> >
> > Regards
> > --
> > Glen Ogilvie
> > Open Systems Specialists
> > Level 1, 162 Grafton Road
> > http://www.oss.co.nz/
> >
> > Ph: +64 9 984 3000
> > Mobile: +64 21 684 146
> > GPG Key: ACED9C17
> >
>

Re: Unsubscribe

[Vijay Kukkala]

Unsubscribe

On Wed, Mar 8, 2017 at 8:06 AM Vijay Kukkala  wrote:

> --
> Best Regards,
> Vijay kukkala
> 318-243-9725
>
-- 
Best Regards,
Vijay kukkala
318-243-9725

offset.storage.filename configuration in kafka-connect-hdfs

[FEI Aggie]

Hi,
I'm running kafka-connect-hdfs 3.1.1.
offset.storage.filename is a required configuration for standalone connector. 
But when I set this parameter in worker configuration file for 
kafka-connect-hdfs with standlone mode:
offset.storage.file.filename=/mnt/data/connect.offsets

It never works. The configured file is not generated.
Anyone know this issue?

Thanks!
Aggie