This is a real problem you describe. Unfortunately adding the
timestamp to the file name won't help the case you describe as
replicas don't directly interact with files they just fetch messages
by offset so there is really no clean way for them to get modification
times from the source broker.
I t
log.retention.bytes can somewhat help. but it is cumbersome to use because
it is a per-topic config for partition limit.
there was an earlier thread regarding global bytes limit. that will work
well for my purpose of avoiding disk full.
https://issues.apache.org/jira/browse/KAFKA-1489
On Thu, Ju
What version of Kafka are your using? Have you tried log.retention.bytes?
Which ever comes first (ttl or bytes total) should do what you are looking
for if I understand you right.
http://kafka.apache.org/documentation.html#brokerconfigs
/***
Joe Stein
Founde
it seems that log retention is purely based on last touch/modified
timestamp. This is undesirable for code push in aws/cloud.
e.g. let's say retention window is 24 hours. disk size is 1 TB. disk util
is 60% (600GB). when new instance comes up, it will fetch log files (600GB)
from peers. those log
How did you start ZK?
Thanks,
Jun
On Wed, Jul 30, 2014 at 10:12 PM, wrote:
> Hi,
>
> Yes I tried. Added CLASSPATH environment variable with value
> D:\Projects\kafka_2.8.0-0.8.1.1\libs\zookeeper-3.3.4.jar and tried to start
> kafka. But still I am getting same error.
>
> Thanks,
> Pradeep Sim
Do you have _any_ logs? i.e., does the consumer log say that it is
rebalancing? If not, what is your log4j level setting?
Joel
On Thu, Jul 31, 2014 at 08:40:45PM +, Srividhya Shanmugam wrote:
> Kafka Team,
>
> In the integration environment Kafka and zookeeper are running under
> supervisio
Set auto.offset.reset to "smallest"
On Thu, Jul 31, 2014 at 08:25:35PM +, Srividhya Shanmugam wrote:
> Kafka Team,
>
> I am using high level consumer API as shown below to read contents from the
> topic.
>
> Properties props = new Properties();
> props.put("zookeeper.connect"
On Thu, Jul 31, 2014 at 11:09:03AM -0700, Kiran Nagasubramanian wrote:
> Thanks for the response Jun and Otis.
>
> Jun, you're right. Only the leaders are reporting. One of the brokers is
> not a leader for any of the partitions. Found the following info on the
> documentation page:
>
> leader i
Kafka Team,
In the integration environment Kafka and zookeeper are running under
supervision. Once in a while when zookeeper and kafka are shut-down and started
back again, the consumers are not able to read the data from the topic. I am
not seeing any exceptions in the log. The consumer offset
Kafka Team,
I am using high level consumer API as shown below to read contents from the
topic.
Properties props = new Properties();
props.put("zookeeper.connect"
,"localhost:2181");
props.put" zookeeper.session.timeout.ms","1
Thanks for the response Jun and Otis.
Jun, you're right. Only the leaders are reporting. One of the brokers is
not a leader for any of the partitions. Found the following info on the
documentation page:
leader is the node responsible for all reads and writes for the given
partition. Each node wil
Can we get the info on targeted release dates for 0.8.2 release and 0.9
release for our planning purposes?
Thanks.
Raja.
On Wed, Jul 30, 2014 at 7:27 PM, Joe Stein wrote:
> The 0.8.2 release will not have the patch inside of it. Trunk already has
> a lot inside of it as a point release. The
I wrote my java code based on kafka unit testing example code. Have you
got it working on a java code?
Regards,
Sathya
On Wed, Jul 30, 2014 at 9:07 PM, Jun Rao wrote:
> Perhaps you can follow some examples in Kafka unit test?
>
> Thanks,
>
> Jun
>
>
> On Tue, Jul 29, 2014 at 7:32 PM, Sathyana
Hi Jim,
Whether to use high level or simple consumer depends on your use case. If
you need to manually manage partition assignments among your consumers, or
you need to commit your offsets elsewhere than ZK, or you do not want auto
rebalancing of consumers upon failures etc, you will use simple co
It is possible to change the replication factor by using the partition
reassignment tool:
http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor
but it requires a manually edited JSON file, KAFKA-1543 tries to make it
easier with a separate command tool.
Guozhang
On T
Hi Sagar,
Its not there in release versions or trunk . Here is the jira
for it https://issues.apache.org/jira/browse/KAFKA-1543.
-Harsha
On Thu, Jul 31, 2014, at 05:22 AM, Sagar Khanna wrote:
> Hi,
>
> >>> re:
> >>> http://grokbase.com/t/kafka/users/13c2rdc2yj/changing-the-
Hi,
>>> re:
>>> http://grokbase.com/t/kafka/users/13c2rdc2yj/changing-the-replication-factor
Is it possible to change the replication factor now or is it still in the
pipeline?
Thanks,
Sagar
Hi,
May also want to share version of Kafka. We have SPM agents for Kafka
0.7.x and 0.8.x and we haven't seen that.
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Wed, Jul 30, 2014 at 10:12 PM, Kiran Nagasubramanian
wr
18 matches
Mail list logo
