This looks at store side of the equation, what's the impact on the load side when the time comes to use this data?
-- Chetan On Tue, Aug 25, 2015 at 8:41 AM, Atri Sharma <[email protected]> wrote: > On 25 Aug 2015 10:34, "Vlad Rozov" <[email protected]> wrote: > > > > I think that the bufferserver should be allowed to use no more than > application specified amount of memory and behavior like linux file cache > will make it difficult to allocate operator/container cache without > reserving too much memory for spikes. > > Sure, agreed. > > My idea is to use *lesser* memory than what is allocated by application > since I am suggesting some level of control over group commits. So I am > thinking of taking the patch you wrote to have it trigger each time buffer > server fills by n units, n being window size. > > If n exceed allocated memory, we can error out. > > Thoughts? > > But I may be wrong and it will be good to have suggested behavior > implemented in a prototype and benchmark prototype performance. > > > > Vlad > > > > > > On 8/24/15 18:24, Atri Sharma wrote: > >> > >> The idea is that if bufferserver dumps *all* pages once it runs out of > >> memory, then it's a huge I/O spike. If it starts paging out once it runs > >> out of memory, then it behaves like a normal cache and further level of > >> paging control can be applied. > >> > >> My idea is that there should be functionality to control the amount of > data > >> that is committed together. This also allows me to 1) define optimal way > >> writes work on my disk 2) allow my application to define locality of > data. > >> For eg I might be performing graph analysis in which a time window's > data > >> consists of sub graph. > >> On 25 Aug 2015 02:46, "Chetan Narsude" <[email protected]> wrote: > >> > >>> The bufferserver writes pages to disk *only when* it runs out of memory > to > >>> hold them. > >>> > >>> Can you elaborate where you see I/O spikes? > >>> > >>> -- > >>> Chetan > >>> > >>> On Mon, Aug 24, 2015 at 12:39 PM, Atri Sharma <[email protected]> > wrote: > >>> > >>>> Folks, > >>>> > >>>> I was wondering if it makes sense to have a functionality in which > >>>> bufferserver writes out data pages to disk in batches defined by > >>>> timeslice/application window. > >>>> > >>>> This will allow flexible workloads and reduce I/O spikes (I understand > >>> > >>> that > >>>> > >>>> we have non-blocking I/O but it still would incur disk head costs). > >>>> > >>>> Thoughts? > >>>> > >>>> -- > >>>> Regards, > >>>> > >>>> Atri > >>>> *l'apprenant* > >>>> > > >
