[ 
https://issues.apache.org/jira/browse/GOBBLIN-1957?focusedWorklogId=890798&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-890798
 ]

ASF GitHub Bot logged work on GOBBLIN-1957:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 15/Nov/23 21:22
            Start Date: 15/Nov/23 21:22
    Worklog Time Spent: 10m 
      Work Description: codecov-commenter commented on PR #3828:
URL: https://github.com/apache/gobblin/pull/3828#issuecomment-1813279350

   ## 
[Codecov](https://app.codecov.io/gh/apache/gobblin/pull/3828?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
 Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Comparison is base 
[(`dd17bed`)](https://app.codecov.io/gh/apache/gobblin/commit/dd17bed4367addd9c68608e2795ffae2d08434b6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
 47.53% compared to head 
[(`b856009`)](https://app.codecov.io/gh/apache/gobblin/pull/3828?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
 45.99%.
   
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3828      +/-   ##
   ============================================
   - Coverage     47.53%   45.99%   -1.54%     
   + Complexity    11035     2179    -8856     
   ============================================
     Files          2156      416    -1740     
     Lines         85377    17984   -67393     
     Branches       9491     2194    -7297     
   ============================================
   - Hits          40581     8272   -32309     
   + Misses        41099     8828   -32271     
   + Partials       3697      884    -2813     
   ```
   
   
   
   </details>
   
   [:umbrella: View full report in Codecov by 
Sentry](https://app.codecov.io/gh/apache/gobblin/pull/3828?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   
   :loudspeaker: Have feedback on the report? [Share it 
here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   




Issue Time Tracking
-------------------

    Worklog Id:     (was: 890798)
    Time Spent: 20m  (was: 10m)

> Add feature to improve ORCWriter buffer sizes with large record sizes
> ---------------------------------------------------------------------
>
>                 Key: GOBBLIN-1957
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1957
>             Project: Apache Gobblin
>          Issue Type: Improvement
>          Components: gobblin-core
>            Reporter: William Lo
>            Assignee: Abhishek Tiwari
>            Priority: Major
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> GobblinORCWriter self tune uses a number of metrics to determine how large 
> their buffers should be for both its own internal buffer used for conversion 
> and the native ORC writer buffer. However when there are very large record 
> sizes (100s of kb) the buffers default max size (e.g. 1000) can still hold a 
> very large amount of data. Observed performance would be hundreds of 
> megabytes to even a gigabyte depending on the configured batch size maximums.
> We want a configuration to impose a maximum buffer max size so that large 
> records in the buffer do not exceed the size of a stripe, so when it is added 
> to the native ORC Writer, the native orc writer should be flushing its 
> records and freeing the memory.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to