[
https://issues.apache.org/jira/browse/ASTERIXDB-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788806#comment-17788806
]
ASF subversion and git services commented on ASTERIXDB-3314:
------------------------------------------------------------
Commit fe5eb2165ce30e75722739a68c3ec2066b81556c in asterixdb's branch
refs/heads/master from Wail Alkowaileet
[ https://gitbox.apache.org/repos/asf?p=asterixdb.git;h=fe5eb2165c ]
[ASTERIXDB-3314][STO] Limit mega leaf node size
- user model changes: yes
- storage format changes: yes
- interface changes: yes
Details:
- Limit mega leaf node sizes to 10MB by default
- Make the max leaf node size configurable
- Local resouce format change to include the max
leaf node size in columnar manager factory
Change-Id: I43965026044543698d73b2f4ad5f8b68b7d3fc99
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/17975
Integration-Tests: Jenkins <[email protected]>
Reviewed-by: Murtadha Hubail <[email protected]>
Tested-by: Murtadha Hubail <[email protected]>
> Reduce buffer cache pressure when operating against columnar datasets
> ---------------------------------------------------------------------
>
> Key: ASTERIXDB-3314
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-3314
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: STO - Storage
> Affects Versions: 0.9.9
> Reporter: Wail Y. Alkowaileet
> Assignee: Wail Y. Alkowaileet
> Priority: Major
> Labels: triaged
> Fix For: 0.9.9
>
>
> When ingesting columnar datasets, the bulkloader and its columnar writers
> rely on the buffer cache to provide the necessary buffers for writing.
> Specifically, the buffer cache provides temporary buffers to the columnar
> writers. However, not all columns require a full 128KB buffers (e.g., sparse
> columns). Instead of using precious buffer cache pages for such columns, we
> should allow column writers to allocate smaller buffers to be used initially.
> In case they needed more space, they can ask the buffer cache for more. This
> approach relieve the pressure against the buffer cache.
>
> Another issue/bug (related to the buffer cache), columnar filters are not
> unpinning their pages when they're done – holding those pages indefinitely
> until the next restart.
>
> A third issue is the merge policy. Currently, we allow merging upto 10
> components in one shot. Reducing this number could also alleviate the
> pressure on the buffer cache.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
