RE: Strange Topic ...

[adrien ruffie]

Hello Ted,


I use Kafka_2.11-1.0.0

And I log file I only have this:

[2018-01-27 23:12:09,545] INFO Shutting down the log cleaner. 
(kafka.log.LogCleaner)
[2018-01-27 23:12:09,545] INFO [kafka-log-cleaner-thread-0]: Shutting down 
(kafka.log.LogCleaner)
[2018-01-27 23:12:09,546] INFO [kafka-log-cleaner-thread-0]: Stopped 
(kafka.log.LogCleaner)
[2018-01-27 23:12:09,546] INFO [kafka-log-cleaner-thread-0]: Shutdown completed 
(kafka.log.LogCleaner)


And the topic appeared this morning, and log log-cleaner.log file, don't log 
anything after 01-27 ...


De : Ted Yu <yuzhih...@gmail.com>
Envoyé : dimanche 4 février 2018 20:37:59
À : users@kafka.apache.org
Objet : Re: Strange Topic ...

Which Kafka version are you using ?
Older versions of kafka (0.10 and prior) had some bugs in the log-cleaner
thread that might sometimes cause it to crash.

Please check the log-cleaner.log file to see if there was some clue.

Cheers

On Sun, Feb 4, 2018 at 11:14 AM, adrien ruffie <adriennolar...@hotmail.fr>
wrote:

> Hello all,
>
>
> I'm a beginner in Kafka and this morning when I try some tests and when
> running this following cmd:
>
> ./bin kafka-topics.sh --zookeeper localhost:2181 --describe
>
>
> I understand my 3 created topic like "customer-topic",
> "streams-plaintext-input", and "streams-wordcount-output"
>
>
> But I already get this following output, why __consumer_offsets have 50
> partitions ! I never created it ... do you know this beavior ?
>
>
> Topic:__consumer_offsetsPartitionCount:50
>  ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=
> compact,compression.type=produ$
> Topic: __consumer_offsets   Partition: 0Leader: 0
>  Replicas: 0 Isr: 0
> Topic: __consumer_offsets   Partition: 1Leader: 0
>  Replicas: 0 Isr: 0
> Topic: __consumer_offsets   Partition: 2Leader: 0
>  Replicas: 0 Isr: 0
> Topic: __consumer_offsets   Partition: 3Leader: 0
>  Replicas: 0 Isr: 0
> Topic: __consumer_offsets   Partition: 4Leader: 0
>  Replicas: 0 Isr: 0
> Topic: __consumer_offsets   Partition: 5Leader: 0
>  Replicas: 0 Isr: 0
> 
> Topic: __consumer_offsets   Partition: 49   Leader: 0
>  Replicas: 0 Isr: 0
>
>
> Topic:customer-topicPartitionCount:1ReplicationFactor:1
>  Configs:
> Topic: customer-topic   Partition: 0Leader: 0   Replicas:
> 0 Isr: 0
> Topic:streams-plaintext-input   PartitionCount:1
> ReplicationFactor:1 Configs:
> Topic: streams-plaintext-input  Partition: 0Leader: 0
>  Replicas: 0 Isr: 0
> Topic:streams-wordcount-output  PartitionCount:1
> ReplicationFactor:1 Configs:cleanup.policy=compact
> Topic: streams-wordcount-output Partition: 0Leader: 0
>  Replicas: 0 Isr: 0
>
>
> Thank and bests regards,
>
> Adrien
>
>
>
>

Re: Strange Topic ...

[naresh Goud]

This is the topic used and created by Kafka internally to store consumer
offsets while use consumer programs running.

Thank you,
Naresh

On Sun, Feb 4, 2018 at 1:38 PM Ted Yu  wrote:

> Which Kafka version are you using ?
> Older versions of kafka (0.10 and prior) had some bugs in the log-cleaner
> thread that might sometimes cause it to crash.
>
> Please check the log-cleaner.log file to see if there was some clue.
>
> Cheers
>
> On Sun, Feb 4, 2018 at 11:14 AM, adrien ruffie 
> wrote:
>
> > Hello all,
> >
> >
> > I'm a beginner in Kafka and this morning when I try some tests and when
> > running this following cmd:
> >
> > ./bin kafka-topics.sh --zookeeper localhost:2181 --describe
> >
> >
> > I understand my 3 created topic like "customer-topic",
> > "streams-plaintext-input", and "streams-wordcount-output"
> >
> >
> > But I already get this following output, why __consumer_offsets have 50
> > partitions ! I never created it ... do you know this beavior ?
> >
> >
> > Topic:__consumer_offsetsPartitionCount:50
> >  ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=
> > compact,compression.type=produ$
> > Topic: __consumer_offsets   Partition: 0Leader: 0
> >  Replicas: 0 Isr: 0
> > Topic: __consumer_offsets   Partition: 1Leader: 0
> >  Replicas: 0 Isr: 0
> > Topic: __consumer_offsets   Partition: 2Leader: 0
> >  Replicas: 0 Isr: 0
> > Topic: __consumer_offsets   Partition: 3Leader: 0
> >  Replicas: 0 Isr: 0
> > Topic: __consumer_offsets   Partition: 4Leader: 0
> >  Replicas: 0 Isr: 0
> > Topic: __consumer_offsets   Partition: 5Leader: 0
> >  Replicas: 0 Isr: 0
> > 
> > Topic: __consumer_offsets   Partition: 49   Leader: 0
> >  Replicas: 0 Isr: 0
> >
> >
> > Topic:customer-topicPartitionCount:1ReplicationFactor:1
> >  Configs:
> > Topic: customer-topic   Partition: 0Leader: 0   Replicas:
> > 0 Isr: 0
> > Topic:streams-plaintext-input   PartitionCount:1
> > ReplicationFactor:1 Configs:
> > Topic: streams-plaintext-input  Partition: 0Leader: 0
> >  Replicas: 0 Isr: 0
> > Topic:streams-wordcount-output  PartitionCount:1
> > ReplicationFactor:1 Configs:cleanup.policy=compact
> > Topic: streams-wordcount-output Partition: 0Leader: 0
> >  Replicas: 0 Isr: 0
> >
> >
> > Thank and bests regards,
> >
> > Adrien
> >
> >
> >
> >
>

Re: Strange Topic ...

[Ted Yu]

Which Kafka version are you using ?
Older versions of kafka (0.10 and prior) had some bugs in the log-cleaner
thread that might sometimes cause it to crash.

Please check the log-cleaner.log file to see if there was some clue.

Cheers

On Sun, Feb 4, 2018 at 11:14 AM, adrien ruffie 
wrote:

> Hello all,
>
>
> I'm a beginner in Kafka and this morning when I try some tests and when
> running this following cmd:
>
> ./bin kafka-topics.sh --zookeeper localhost:2181 --describe
>
>
> I understand my 3 created topic like "customer-topic",
> "streams-plaintext-input", and "streams-wordcount-output"
>
>
> But I already get this following output, why __consumer_offsets have 50
> partitions ! I never created it ... do you know this beavior ?
>
>
> Topic:__consumer_offsetsPartitionCount:50
>  ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=
> compact,compression.type=produ$
> Topic: __consumer_offsets   Partition: 0Leader: 0
>  Replicas: 0 Isr: 0
> Topic: __consumer_offsets   Partition: 1Leader: 0
>  Replicas: 0 Isr: 0
> Topic: __consumer_offsets   Partition: 2Leader: 0
>  Replicas: 0 Isr: 0
> Topic: __consumer_offsets   Partition: 3Leader: 0
>  Replicas: 0 Isr: 0
> Topic: __consumer_offsets   Partition: 4Leader: 0
>  Replicas: 0 Isr: 0
> Topic: __consumer_offsets   Partition: 5Leader: 0
>  Replicas: 0 Isr: 0
> 
> Topic: __consumer_offsets   Partition: 49   Leader: 0
>  Replicas: 0 Isr: 0
>
>
> Topic:customer-topicPartitionCount:1ReplicationFactor:1
>  Configs:
> Topic: customer-topic   Partition: 0Leader: 0   Replicas:
> 0 Isr: 0
> Topic:streams-plaintext-input   PartitionCount:1
> ReplicationFactor:1 Configs:
> Topic: streams-plaintext-input  Partition: 0Leader: 0
>  Replicas: 0 Isr: 0
> Topic:streams-wordcount-output  PartitionCount:1
> ReplicationFactor:1 Configs:cleanup.policy=compact
> Topic: streams-wordcount-output Partition: 0Leader: 0
>  Replicas: 0 Isr: 0
>
>
> Thank and bests regards,
>
> Adrien
>
>
>
>

Re: Strange topic-corruption issue?

[Steve Miller]

   Odd -- I copied and pasted what you'd asked me to run:

/opt/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 
.log

and I just re-ran it and the output looks the same as what I'd put up for 
people to grab.

   I also ran:

/opt/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --deep-iteration 
--files .log

and the output is identical (I diffed it to be sure).

   The publisher is publishing with compression turned off, though; when we had 
it turned on, I remember seeing some references to Snappy in there.  I'd turned 
compression off on the theory that (at the time) we had one thing producing in 
a way that caused corruption and one thing that was producing using 
compression, and maybe they were linked (but turning compression off didn't 
help).

-Steve

On Fri, Aug 15, 2014 at 04:00:33PM -0700, Jun Rao wrote:
 What's in there seems to be still the output for deep iteration. For
 shallow iteration, the compression codec for each message should be Snappy.
 
 Thanks,
 
 Jun
 
 
 On Fri, Aug 15, 2014 at 5:27 AM, Steve Miller st...@idrathernotsay.com
 wrote:
 
 Oh, yeah, sorry about that.  I threw a copy of that up at:
 
  https://newbie.idrathernotsay.com/full.txt.gz
 
  (you'll get a cert error, for the four times I put something on the home
  website each year, I didn't feel like getting a real cert (-: ).
 
 If that doesn't work I'm sure I can figure something else out.
 
  -Steve
 
  On Thu, Aug 14, 2014 at 05:04:29PM -0700, Neha Narkhede wrote:
   Apache doesn't allow attachments. Could you send maybe a pastebin or
   something?