+1 Some information on how this will be tested would be useful.
On 4/23/15 9:33 AM, "Jay Kreps" <jay.kr...@gmail.com> wrote: >Yeah if we understand the optimal policy for a config we always want to >set >it automatically. In this case I don't think we do yet, but down the road >that could be nice. I think for now we should consider this option >experimental to give people a chance to try it out. > >-Jay > >On Wed, Apr 22, 2015 at 7:32 PM, Honghai Chen <honghai.c...@microsoft.com> >wrote: > >> Hi Roshan, >> Use the 'auto' value maybe will break the rule and mess up the >> configuration. @Jay, any thoughts? >> >> Thanks, Honghai Chen >> >> -----Original Message----- >> From: Sriharsha Chintalapani [mailto:harsh...@fastmail.fm] >> Sent: Thursday, April 23, 2015 6:27 AM >> To: dev@kafka.apache.org; Roshan Naik >> Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume >> performance under windows and some old Linux file system >> >> +1 (non-binding). >> >> -- >> Harsha >> >> >> On April 22, 2015 at 2:52:12 PM, Roshan Naik (ros...@hortonworks.com) >> wrote: >> >> I see that it is safe to keep it this off by default due to some >>concerns. >> Eventually, for settings such as this whose 'preferred' value is >>platform >> specific (or based on other criteria), it might be worth considering >> having a default value that is not a constant but an 'auto' value .. >>When >> kafka boots up it can automatically use the preferred value. Ofcourse it >> would have to documented as to what auto means for a given platform. >> >> -roshan >> >> >> On 4/22/15 1:21 PM, "Jakob Homan" <jgho...@gmail.com> wrote: >> >> >+1. This is an important performance fix for Windows-based clusters. >> > >> >-Jakob >> > >> >On 22 April 2015 at 03:25, Honghai Chen <honghai.c...@microsoft.com> >> >wrote: >> >> Fix the issue Sriram mentioned. Code review and jira/KIP updated. >> >> >> >> Below are detail description for the scenarios: >> >> 1.If do clear shutdown, the last log file will be truncated to its >> >>real size since the close() function of FileMessageSet will call >>trim(), >> >> 2.If crash, then when restart, will go through the process of >> >>recover() and the last log file will be truncate to its real size, >>(and >> >>the position will be moved to end of the file) >> >> 3.When service start and open existing file >> >> a.Will run the LogSegment constructor which has NO parameter >> >>"preallocate", >> >> b.Then in FileMessageSet, the "end" in FileMessageSet will be >> >>Int.MaxValue, and then >> >>"channel.position(math.min(channel.size().toInt, end))" will make the >> >>position be end of the file, >> >> c.If recover needed, the recover function will truncate file to end >>of >> >>valid data, and also move the position to it, >> >> >> >> 4.When service running and need create new log segment and new >> >>FileMessageSet >> >> >> >> a.If preallocate = truei.the "end" in FileMessageSet will be 0, the >> >>file size will be "initFileSize", and then >> >>"channel.position(math.min(channel.size().toInt, end))" will make the >> >>position be 0, >> >> >> >> b.Else if preallocate = falsei.backward compatible, the "end" in >> >>FileMessageSet will be Int.MaxValue, the file size will be "0", and >> >>then "channel.position(math.min(channel.size().toInt, end))" will make >> >>the position be 0, >> >> >> >> >> >> >> >>https://cwiki.apache.org/confluence/display/KAFKA/KIP-20+-+Enable+log+pre >> >>>>allocate+to+improve+consume+performance+under+windows+and+some+old+Linu >>>>x+ >> >>file+system >> >> https://issues.apache.org/jira/browse/KAFKA-1646 >> >> https://reviews.apache.org/r/33204/diff/2/ >> >> >> >> Thanks, Honghai Chen >> >> http://aka.ms/kafka >> >> http://aka.ms/manifold >> >> >> >> -----Original Message----- >> >> From: Honghai Chen >> >> Sent: Wednesday, April 22, 2015 11:12 AM >> >> To: dev@kafka.apache.org >> >> Subject: RE: [DISCUSS] KIP 20 Enable log preallocate to improve >>consume >> >>performance under windows and some old Linux file system >> >> >> >> Hi Sriram, >> >> One sentence of code missed, will update code review board and >> >>KIP soon. >> >> For LogSegment and FileMessageSet, must use different >> >>constructor function for existing file and new file, then the code " >> >>channel.position(math.min(channel.size().toInt, end)) " will make sure >> >>the position at end of existing file. >> >> >> >> Thanks, Honghai Chen >> >> >> >> -----Original Message----- >> >> From: Jay Kreps [mailto:jay.kr...@gmail.com] >> >> Sent: Wednesday, April 22, 2015 5:22 AM >> >> To: dev@kafka.apache.org >> >> Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve >>consume >> >>performance under windows and some old Linux file system >> >> >> >> My understanding of the patch is that clean shutdown truncates the >>file >> >>back to it's true size (and reallocates it on startup). Hard crash is >> >>handled by the normal recovery which should truncate off the empty >> >>portion of the file. >> >> >> >> On Tue, Apr 21, 2015 at 10:52 AM, Sriram Subramanian < >> >>srsubraman...@linkedin.com.invalid> wrote: >> >> >> >>> Could you describe how recovery works in this mode? Say, we had a >>250 >> >>> MB preallocated segment and we wrote till 50MB and crashed. Till >>what >> >>> point do we recover? Also, on startup, how is the append end pointer >> >>> set even on a clean shutdown? How does the FileChannel end position >> >>> get set to 50 MB instead of 250 MB? The existing code might just >>work >> >>> for it but explaining that would be useful. >> >>> >> >>> On 4/21/15 9:40 AM, "Neha Narkhede" <n...@confluent.io> wrote: >> >>> >> >>> >+1. I've tried this on Linux and it helps reduce the spikes in >>append >> >>> >+(and >> >>> >hence producer) latency for high throughput writes. I am not >>entirely >> >>> >sure why but my suspicion is that in the absence of preallocation, >> >>> >you see spikes writes need to happen faster than the time it takes >> >>> >Linux to allocate the next block to the file. >> >>> > >> >>> >It will be great to see some performance test results too. >> >>> > >> >>> >On Tue, Apr 21, 2015 at 9:23 AM, Jay Kreps <jay.kr...@gmail.com> >> >>>wrote: >> >>> > >> >>> >> I'm also +1 on this. The change is quite small and may actually >> >>> >>help perf on Linux as well (we've never tried this). >> >>> >> >> >>> >> I have a lot of concerns on testing the various failure >>conditions >> >>> >> but I think since it will be off by default the risk is not too >> >>>high. >> >>> >> >> >>> >> -Jay >> >>> >> >> >>> >> On Mon, Apr 20, 2015 at 6:58 PM, Honghai Chen >> >>> >><honghai.c...@microsoft.com> >> >>> >> wrote: >> >>> >> >> >>> >> > I wrote a KIP for this after some discussion on KAFKA-1646. >> >>> >> > https://issues.apache.org/jira/browse/KAFKA-1646 >> >>> >> > >> >>> >> > >> >>> >> >> >>> >> >> >>> >>https://cwiki.apache.org/confluence/display/KAFKA/KIP-20+-+Enable+log+ >> >>> pre >> >>> >> >>>>>>>allocate+to+improve+consume+performance+under+windows+and+some+old+L >>>>>>>in >> >>>>>ux+ >> >>> >>file+system >> >>> >> > The RB is here: https://reviews.apache.org/r/33204/diff/ >> >>> >> > >> >>> >> > Thanks, Honghai >> >>> >> > >> >>> >> > >> >>> >> >> >>> > >> >>> > >> >>> > >> >>> >-- >> >>> >Thanks, >> >>> >Neha >> >>> >> >>> >> >>
