Hey Stevo,
I think ConsumerRecords only contains the partitions which had messages.
Would you mind creating a jira for the feature request? You're welcome to
submit a patch as well.
-Jason
On Tue, Jul 21, 2015 at 2:27 AM, Stevo Slavić ssla...@gmail.com wrote:
Hello Apache Kafka community,
Thank you, Nicolas!
On Tue, Jul 21, 2015 at 10:46 AM, Nicolas Phung nicolas.ph...@gmail.com
wrote:
Yes indeed.
# A comma seperated list of directories under which to store log files
log.dirs=/var/lib/kafka
You can put several disk/partitions too.
Regards,
On Tue, Jul 21, 2015 at 4:37
This is a known issue. There are a few relevant JIRAs and a KIP:
https://issues.apache.org/jira/browse/KAFKA-1788
https://issues.apache.org/jira/browse/KAFKA-2120
https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient
-Ewen
On Tue, Jul 21, 2015 at 7:05
On Tue, Jul 21, 2015 at 2:38 AM, Stevo Slavić ssla...@gmail.com wrote:
Hello Apache Kafka community,
I find new consumer poll/seek javadoc a bit confusing. Just by reading docs
I'm not sure what the outcome will be, what is expected in following
scenario:
- kafkaConsumer is instantiated
Since you mentioned consumer groups, I'm assuming you're using the high
level consumer? Do you have auto.commit.enable set to true?
It sounds like when you start up you are always getting the
auto.offset.reset behavior, which indicates you don't have any offsets
committed. By default, that
Hi,
We've been using Kafka for a couple of months, and now we're trying to to
write a Simple application using the ConsumerGroup to fully understand
Kafka.
Having the producer continually writing data, our consumer occasionally
needs to be restarted. However, once the program is brought back up,
Hi all,
If I wanted to write my own partitioner, all I would need to do is write a
class that extends Partitioner and override the partition function,
correct? I am currently doing so, at least, with a class in the package
'services', yet when I use:
properties.put(partitioner.class,
Hi,
Are you using the latest trunk for Producer API?. Did you implement the
interface here
https://cwiki.apache.org/confluence/display/KAFKA/KIP-+22+-+Expose+a+Partitioner+interface+in+the+new+producer
--
Harsha
On July 21, 2015 at 2:27:05 PM, JIEFU GONG (jg...@berkeley.edu) wrote:
Hi
Sriharsha, thanks for your response. I'm using version 0.8.2, and I am
implementing kafka.producer.Partitioner.
I noticed that in the latest trunk the line I specified above is replaced
with:
this.partitioner = config.getConfiguredInstance(ProducerConfig.
PARTITIONER_CLASS_CONFIG,
If you are using the new producer api from kafka 0.8.2 there is no pluggable
partitioner in it for this you need to use the latest trunk. But in 0.8.2 if
you are using old producer code you can implement a pluggable partitioner
Hi,
The document about zookeeper.connect on Broker Configs says that
Note that you must create this path yourself prior to starting the broker,
but it seems the broker creates the path automatically on start up
(maybe related issue: https://issues.apache.org/jira/browse/KAFKA-404 ).
So the
On Jul 21, 2015, at 9:15 AM, Ewen Cheslack-Postava e...@confluent.io wrote:
On Tue, Jul 21, 2015 at 2:38 AM, Stevo Slavić ssla...@gmail.com wrote:
Hello Apache Kafka community,
I find new consumer poll/seek javadoc a bit confusing. Just by reading docs
I'm not sure what the outcome
Hi Nicolas,
From my experience there are only two ways out:
1) wait for retention time to pass, so data gets deleted (this is usually
unacceptable)
2) trace offset of corrupt message on all affected subscriptions and skip
this message by overwriting it (offset+1)
Problem is, that when
Hello,
I'm using Confluent Kafka (0.8.2.0-cp). When I'm trying to process message
from my Kafka topic with Spark Streaming, I've got the following error :
kafka.message.InvalidMessageException: Message is corrupt (stored crc =
3561357254, computed crc = 171652633)
at
Thanks all for fast feedback!
It's great news if that aspect is improved as well in new HLC. I will test
and get back with any related findings.
Kind regards,
Stevo Slavic.
On Mon, Jul 20, 2015 at 9:57 PM, Guozhang Wang wangg...@gmail.com wrote:
Hi Stevo,
I am still not very clear on your
Hello Apache Kafka community,
New HLC poll returns ConsumerRecords.
Do ConsumerRecords contain records for every partition that HLC is actively
subscribed on for every poll request, or does it contain only records for
partitions which had messages and which were retrieved in poll request?
If
Hello Apache Kafka community,
I find new consumer poll/seek javadoc a bit confusing. Just by reading docs
I'm not sure what the outcome will be, what is expected in following
scenario:
- kafkaConsumer is instantiated with auto-commit off
- kafkaConsumer.subscribe(someTopic)
-
Hello Apache Kafka community,
Just noticed that :
- message is successfully published using new 0.8.2.1 producer
- and then Kafka is stopped
next attempt to publish message using same instance of new producer hangs
forever, and following stacktrace gets logged repeatedly:
[WARN ]
Just wanna make sure, in server.properties, the configuration
log.dirs=/tmp/kafka-logs
specifies the directory of where the log (data) stores, right?
If I want the data to be saved elsewhere, this is the configuration I need
to change, right?
Thanks for answering.
best,
19 matches
Mail list logo
