[
https://issues.apache.org/jira/browse/TEZ-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948912#comment-13948912
]
Hitesh Shah commented on TEZ-932:
---------------------------------
Overall comment
- there should be a big enough paragraph at the start explaining what this
class does and how it does it.
- at this point, for a user trying to use this, there is hardly any
information on how to configure and use it. From that point of view, it seems a
new user will never try to use this.
- it should be okay to go once documentation is addressed given that it can
be considered somewhat "experimental" and also as its not the default option in
use - have not looked too deeply into correctness though
{code}
+ public static final String
TEZ_TASK_SCALE_MEMORY_ADDITIONAL_RESERVATION_FRACTION_PER_IO = TEZ_TASK_PREFIX
+ + "scale.memory.additional.reservation.fraction.per-io";
+
+ /**
+ * Max cumulative total reservation for additional IOs.
+ */
+ public static final String
TEZ_TASK_SCALE_MEMORY_ADDITIONAL_RESERVATION_FRACTION_MAX_TOTAL_LIMIT =
TEZ_TASK_PREFIX
+ + "scale.memory.additional reservation.fraction.max-total-limit";
{code}
- maybe slightly shorter properties if possible?
s/RuntimeLibraryInitialMemoryAllocator/InitialMemoryAllocatorImpl/ ? or just
BasicInitialMemoryAllocator ?
{code}
+ // Fall back to regular scaling. e.g. BROADCAST : SHUFFLE = 0:1 .
+ // i.e. if Shuffle present, Broadcast gets nothing, but otherwise it
+ // should get an allocation
{code}
- wasnt there recent code that would start throwing an error if received 0
for available memory ?
bq. if (reserveFraction >= 1.0d) {
- why is this not an error where the task is failed due to misconfiguration?
bq. TezJobConfig.TEZ_TASK_SCALE_MEMORY_WEIGHTED_RATIOS
- this does not seem right. What is the format? Seems to have 5 numbers
which are set without any clue as to what they map to.
> Add a type aware Initial memory distributor
> -------------------------------------------
>
> Key: TEZ-932
> URL: https://issues.apache.org/jira/browse/TEZ-932
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: TEZ-932.1.txt, TEZ-932.2.txt
>
>
> The current memory scaler has not idea about the requirements of the
> underlying components (broadcast, shfufle etc). This jira is to add a scaler
> which can take the types into consideration.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
