-Original Message-
From: Jun Rao [mailto:j...@confluent.io]
Sent: Tuesday, May 5, 2015 12:51 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve
consume performance under windows and some old Linux file system
This seems
, April 24, 2015 12:57 AM
To: dev@kafka.apache.org
Cc: Roshan Naik
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve
consume performance under windows and some old Linux file system
+1
Some information on how this will be tested would be useful.
On 4/23/15 9
Chen
-Original Message-
From: Jun Rao [mailto:j...@confluent.io]
Sent: Tuesday, May 5, 2015 12:51 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve
consume performance under windows and some old Linux file system
-
From: Jun Rao [mailto:j...@confluent.io]
Sent: Wednesday, May 6, 2015 5:39 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux file system
Thanks. Could you updated the wiki? Also, commented
Chintalapani [mailto:harsh...@fastmail.fm]
Sent: Thursday, April 23, 2015 6:27 AM
To: dev@kafka.apache.org; Roshan Naik
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to
improve
consume performance under windows and some old Linux file
system
+1 (non-binding
@kafka.apache.org
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux file system
Thanks. Could you updated the wiki? Also, commented on the jira.
Jun
On Tue, May 5, 2015 at 12:48 AM, Honghai Chen
honghai.c
, May 6, 2015 5:39 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux file system
Thanks. Could you updated the wiki? Also, commented on the jira.
Jun
On Tue, May 5, 2015 at 12:48 AM, Honghai Chen
@kafka.apache.org
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux file system
Thanks. Could you updated the wiki? Also, commented on the jira.
Jun
On Tue, May 5, 2015 at 12:48 AM, Honghai Chen honghai.c...@microsoft.com
wrote:
Use
Subramanian [mailto:srsubraman...@linkedin.com.INVALID]
Sent: Friday, April 24, 2015 12:57 AM
To: dev@kafka.apache.org
Cc: Roshan Naik
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve
consume performance under windows and some old Linux file system
+1
Some information on how
...@confluent.io]
Sent: Tuesday, May 5, 2015 12:51 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux file system
This seems similar to what's in
https://issues.apache.org/jira/browse/KAFKA-1065.
Also
@kafka.apache.org
Cc: Roshan Naik
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux file system
+1
Some information on how this will be tested would be useful.
On 4/23/15 9:33 AM, Jay Kreps jay.kr...@gmail.com wrote:
Yeah if we understand
: Thursday, April 23, 2015 6:27 AM
To: dev@kafka.apache.org; Roshan Naik
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux file system
+1 (non-binding).
--
Harsha
On April 22, 2015 at 2:52:12 PM, Roshan Naik (ros
: [DISCUSS] KIP 20 Enable log preallocate to improve
consume
performance under windows and some old Linux file system
My understanding of the patch is that clean shutdown truncates the
file
back to it's true size (and reallocates it on startup). Hard crash is
handled by the normal recovery which
@kafka.apache.org; Roshan Naik
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux file system
+1 (non-binding).
--
Harsha
On April 22, 2015 at 2:52:12 PM, Roshan Naik (ros...@hortonworks.com) wrote:
I see that it is safe to keep it this off
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux file system
My understanding of the patch is that clean shutdown truncates the file
back to it's true size (and reallocates it on startup). Hard
/
Thanks, Honghai Chen
http://aka.ms/kafka
http://aka.ms/manifold
-Original Message-
From: Honghai Chen
Sent: Wednesday, April 22, 2015 11:12 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux
Subject: RE: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows and some old Linux file system
Hi Sriram,
One sentence of code missed, will update code review board and KIP soon.
For LogSegment and FileMessageSet, must use different constructor
Could you describe how recovery works in this mode? Say, we had a 250 MB
preallocated segment and we wrote till 50MB and crashed. Till what point
do we recover? Also, on startup, how is the append end pointer set even on
a clean shutdown? How does the FileChannel end position get set to 50 MB
My understanding of the patch is that clean shutdown truncates the file
back to it's true size (and reallocates it on startup). Hard crash is
handled by the normal recovery which should truncate off the empty portion
of the file.
On Tue, Apr 21, 2015 at 10:52 AM, Sriram Subramanian
the
position at end of existing file.
Thanks, Honghai Chen
-Original Message-
From: Jay Kreps [mailto:jay.kr...@gmail.com]
Sent: Wednesday, April 22, 2015 5:22 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
performance under windows
I'm also +1 on this. The change is quite small and may actually help perf
on Linux as well (we've never tried this).
I have a lot of concerns on testing the various failure conditions but I
think since it will be off by default the risk is not too high.
-Jay
On Mon, Apr 20, 2015 at 6:58 PM,
+1. I've tried this on Linux and it helps reduce the spikes in append (and
hence producer) latency for high throughput writes. I am not entirely sure
why but my suspicion is that in the absence of preallocation, you see
spikes writes need to happen faster than the time it takes Linux to
allocate
I wrote a KIP for this after some discussion on KAFKA-1646.
https://issues.apache.org/jira/browse/KAFKA-1646
https://cwiki.apache.org/confluence/display/KAFKA/KIP-20+-+Enable+log+preallocate+to+improve+consume+performance+under+windows+and+some+old+Linux+file+system
The RB is here:
+1 (non-binding)
Sure, makes sense :)
Just make sure the doc for the config includes something like If you
are using Kafka on Windows, you probably want to set it to true, so
people will know how to use it without looking for the JIRA.
On Mon, Apr 20, 2015 at 6:58 PM, Honghai Chen
24 matches
Mail list logo
