I think that the bufferserver should be allowed to use no more than
application specified amount of memory and behavior like linux file
cache will make it difficult to allocate operator/container cache
without reserving too much memory for spikes. But I may be wrong and it
will be good to have suggested behavior implemented in a prototype and
benchmark prototype performance.
Vlad
On 8/24/15 18:24, Atri Sharma wrote:
The idea is that if bufferserver dumps *all* pages once it runs out of
memory, then it's a huge I/O spike. If it starts paging out once it runs
out of memory, then it behaves like a normal cache and further level of
paging control can be applied.
My idea is that there should be functionality to control the amount of data
that is committed together. This also allows me to 1) define optimal way
writes work on my disk 2) allow my application to define locality of data.
For eg I might be performing graph analysis in which a time window's data
consists of sub graph.
On 25 Aug 2015 02:46, "Chetan Narsude" <[email protected]> wrote:
The bufferserver writes pages to disk *only when* it runs out of memory to
hold them.
Can you elaborate where you see I/O spikes?
--
Chetan
On Mon, Aug 24, 2015 at 12:39 PM, Atri Sharma <[email protected]> wrote:
Folks,
I was wondering if it makes sense to have a functionality in which
bufferserver writes out data pages to disk in batches defined by
timeslice/application window.
This will allow flexible workloads and reduce I/O spikes (I understand
that
we have non-blocking I/O but it still would incur disk head costs).
Thoughts?
--
Regards,
Atri
*l'apprenant*
