[
https://issues.apache.org/jira/browse/ORC-21?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955534#comment-14955534
]
ASF GitHub Bot commented on ORC-21:
-----------------------------------
GitHub user asandryh opened a pull request:
https://github.com/apache/orc/pull/10
Fixed ORC-21: Add functionality to estimate dynamic memory requirements
An upper bound on memory requirements is provided by two components:
- Reader::memoryUse() returns an upper bound on its memory needs. It
depends on the file and columns read.
- ColumnBatch::memoryUse() returns an upper bound on its memory needs. It
depends on the file, columns, and number of rows read.
The new utility FileMemory.cc compares estimated and actual memory usage.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/asandryh/orc master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/orc/pull/10.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10
----
commit c31b3ed204ba01ad3835d963ec2eb8343fd2981e
Author: Aliaksei Sandryhaila <[email protected]>
Date: 2015-07-08T15:06:57Z
Added a stream block size parameter to ReaderOptions to change the size of
InputStreams' buffers.
commit 7114d6e89b7f96762b07479b397298edb8412a8c
Author: Aliaksei Sandryhaila <[email protected]>
Date: 2015-07-08T15:14:18Z
Corrected type casting.
commit d40382662e9b564354160fa8a0d833ab5c699a08
Author: Aliaksei Sandryhaila <[email protected]>
Date: 2015-07-24T20:17:26Z
Revert "Corrected type casting."
This reverts commit 7114d6e89b7f96762b07479b397298edb8412a8c.
commit 5992cc95d63bbcab362e685e65d0748277c7582d
Author: Aliaksei Sandryhaila <[email protected]>
Date: 2015-07-24T20:17:47Z
Revert "Added a stream block size parameter to ReaderOptions to change the
size of InputStreams' buffers."
This reverts commit c31b3ed204ba01ad3835d963ec2eb8343fd2981e.
commit fe2714d535c1acaccf1bbf25715f2e1934d76fcb
Author: Aliaksei Sandryhaila <[email protected]>
Date: 2015-07-24T20:23:42Z
Merge remote-tracking branch 'upstream/master'
commit 51ec0491dc294101d377045b3665786cb3fe435d
Author: Aliaksei Sandryhaila <[email protected]>
Date: 2015-07-29T16:37:13Z
Merge remote-tracking branch 'upstream/master'
commit dd3b8212ba6fa8b88ccec876b488942abfec437e
Author: Aliaksei Sandryhaila <[email protected]>
Date: 2015-09-18T21:38:11Z
Work in progress: added [incomplete] functionality for memory estimation.
commit 0277eb695c8f61a72ec192c8fc57d17e02d1c34b
Author: Aliaksei Sandryhaila <[email protected]>
Date: 2015-09-18T21:41:53Z
Work in progress: adding memory estimation functionality.
commit d051d0e73ccee4ee2113f8de625b895117937db6
Author: Aliaksei Sandryhaila <[email protected]>
Date: 2015-09-18T21:49:45Z
Merge remote-tracking branch 'upstream/master'
commit 822a9da76370b97e6231693c6539d677175b3193
Author: Aliaksei Sandryhaila <[email protected]>
Date: 2015-09-23T15:46:37Z
[Work in progress] Remove debugging code, modify unit tests.
commit 03795cd26c637e0cc5c722463bbdebc11000f09d
Author: Aliaksei Sandryhaila <[email protected]>
Date: 2015-10-12T13:41:15Z
Merge remote-tracking branch 'upstream/master'
commit 141b56a837e665c2176fc528b6c619e663872776
Author: Aliaksei Sandryhaila <[email protected]>
Date: 2015-10-13T18:47:01Z
Fixed ORC-21: Added functionality to estimate dynamic memory requirements.
commit f5f7ef340e3b20db6d3385be186da54ac49beb41
Author: Aliaksei Sandryhaila <[email protected]>
Date: 2015-10-13T18:50:45Z
Removed debug code.
commit 2c92d49990e76b6a97aa4be7f0400122333e0810
Author: Aliaksei Sandryhaila <[email protected]>
Date: 2015-10-13T19:38:26Z
Minor code clean-up.
commit 219b93a42694870547fcc7f5f5682dd5efb4d59e
Author: Aliaksei Sandryhaila <[email protected]>
Date: 2015-10-13T19:47:49Z
More clean-up.
----
> Add functionality to estimate memory footprint
> ----------------------------------------------
>
> Key: ORC-21
> URL: https://issues.apache.org/jira/browse/ORC-21
> Project: Orc
> Issue Type: Task
> Reporter: Aliaksei Sandryhaila
> Assignee: Aliaksei Sandryhaila
>
> ORC library allocates multiple large buffers to read and materialize ORC
> files. For stability of applications that use the library, it may be
> desirable to have an estimate (preferably, a tight upper bound) of a memory
> footprint.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
