[
https://issues.apache.org/jira/browse/BEAM-10212?focusedWorklogId=642924&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642924
]
ASF GitHub Bot logged work on BEAM-10212:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 27/Aug/21 17:12
Start Date: 27/Aug/21 17:12
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #15300:
URL: https://github.com/apache/beam/pull/15300#issuecomment-907350175
> One benefit of the static limit versus percentage is that if OOMs are
caused by caching too much (or cache memory tracking not working as well as
expected), the static limit allows for the machine size to be increased to
avoid the OOMs as the cache will use the same amount of memory. If the max
cache size is automatically increased proportional to the worker shape, the
OOMs will just continue possibly negating VM right-sizing or manual size tuning.
>
> Perhaps instead of a static value or a static percentage we could do a
combination:
> static_value + % of machine size
> Then we don't scale entirely with the machine size but also will use a
larger cache by default for larger machines.
VMR and/or users should specify a machine with _more memory_ and not one
with _more cores and memory_ since we want to increase the amount of memory per
core. This would still allow a pure percentage based model to work since a
machine with _more memory_ will still have a greater amount of memory that is
not dedicated to the cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 642924)
Time Spent: 14h 20m (was: 14h 10m)
> Add support for Java SDK harness state caching
> ----------------------------------------------
>
> Key: BEAM-10212
> URL: https://issues.apache.org/jira/browse/BEAM-10212
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-harness
> Reporter: Luke Cwik
> Assignee: Anthony Zhu
> Priority: P3
> Labels: Clarified, portability
> Time Spent: 14h 20m
> Remaining Estimate: 0h
>
> Tech spec:
> [https://docs.google.com/document/d/1BOozW0bzBuz4oHJEuZNDOHdzaV5Y56ix58Ozrqm2jFg/edit#heading=h.7ghoih5aig5m]
> Relevant document:
> [https://docs.google.com/document/d/1ltVqIW0XxUXI6grp17TgeyIybk3-nDF8a0-Nqw-s9mY/edit#|https://docs.google.com/document/d/1ltVqIW0XxUXI6grp17TgeyIybk3-nDF8a0-Nqw-s9mY/edit]
> Mailing list link:
> [https://lists.apache.org/thread.html/caa8d9bc6ca871d13de2c5e6ba07fdc76f85d26497d95d90893aa1f6@%3Cdev.beam.apache.org%3E]
>
> See https://issues.apache.org/jira/browse/BEAM-5428 and
> https://issues.apache.org/jira/browse/BEAM-8298 for Python implementation
> details.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
