[
https://issues.apache.org/jira/browse/GOBBLIN-1918?focusedWorklogId=881797&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-881797
]
ASF GitHub Bot logged work on GOBBLIN-1918:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 25/Sep/23 22:24
Start Date: 25/Sep/23 22:24
Worklog Time Spent: 10m
Work Description: Will-Lo merged PR #3787:
URL: https://github.com/apache/gobblin/pull/3787
Issue Time Tracking
-------------------
Worklog Id: (was: 881797)
Time Spent: 1h (was: 50m)
> Optimize smart resizing for ORC Writer converter buffer
> -------------------------------------------------------
>
> Key: GOBBLIN-1918
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1918
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-core
> Reporter: William Lo
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The GobblinOrcWriter contains a converter and a buffer rowbatch. The buffer
> holds the converted Avro -> Orc records before adding them to the native orc
> writer.
> Since it can contain multiple records, it constantly needs to resize the
> columns of the rowbatch in order to hold multiple records. This problem
> affects both performance and memory when resizing is done either too often
> (enlarge factor is too low) or not often enough (enlarge factor is too high
> and thus the buffer dominates the container memory).
> Because there is a bounded number of records that can persist in the buffer
> before getting flushed, we want to reduce the aggressiveness of the resizing
> algorithm the more records that have been processed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
