[
https://issues.apache.org/jira/browse/YARN-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231658#comment-17231658
]
Szilard Nemeth commented on YARN-7200:
--------------------------------------
Hi [~akshink],
Checked the call hierarchy of
org.apache.hadoop.yarn.sls.scheduler.SchedulerMetrics#tearDown, and to me it
looks good because it is only getting invoked from these 2 methods:
{code:java}
SLSFairScheduler.serviceStop() (org.apache.hadoop.yarn.sls.scheduler)
SLSCapacityScheduler.serviceStop() (org.apache.hadoop.yarn.sls.scheduler)
{code}
So, the array closing character is only placed when the scheduler (instance of
AbstractService) is getting stopped, meaning that the serviceStop method is
invoked.
*Also played around with your patch with the following way:*
1. Modified the parameterized class called TestSLSStreamAMSynth to only
execute tests with CS.
2. Added some printouts to SchedulerMetrics.
3. Launched
org.apache.hadoop.yarn.sls.TestSLSStreamAMSynth#testSimulatorRunning.
See the attached patch file and test logs in the zip for details.
I can only see "***teardown: Closing array in json" in the test log output,
which is coming from
org.apache.hadoop.yarn.sls.scheduler.SchedulerMetrics#tearDown.
I also added a log string to
org.apache.hadoop.yarn.sls.scheduler.SchedulerMetrics#closeMetricsWriter, with
String: "**closeMetricsWriter: Closing array in json", but I can't see it in
the output, so this is suspicious.
Have you been able to reproduce an issue with the original code, so that the
array-closing character is misplaced in the realtimetrack.json?
If you check the attachment called 'realtimetrack-intellij-debugging.json' (in
the zip), you can see that the array-closing char is misplaced, but this only
happened when I was debugging in intellij so I wouldn't count it as an issue.
*Could you please justify the following?*
*1. Why the method called
org.apache.hadoop.yarn.sls.scheduler.SchedulerMetrics#closeMetricsWriter is not
getting called from this testcase?*
What was your testing methodology?
Maybe the testcase is different than regular launch of SLS from the CLI, but I
think it should invoke this method, anyway.
*2. What was the issue with the
org.apache.hadoop.yarn.sls.scheduler.SchedulerMetrics#tearDown method?*
I think it's a good place to close the array in the json file.
Considering the method calls to
org.apache.hadoop.yarn.sls.scheduler.SchedulerMetrics#metricsLogBW in
SchedulerMetrics, I think adding the array-closing should be in one centralized
place, like it has been.
However, there can be situations where someone kills the JVM process and the
regular serviceStop methods won't run for some services, so you could add a
shutdown hook to cover such a case, but I think it would be an overkill.
[~pbacsko], [~shuzirra] Please chime in.
> SLS generates a realtimetrack.json file but that file is missing the closing
> ']'
> --------------------------------------------------------------------------------
>
> Key: YARN-7200
> URL: https://issues.apache.org/jira/browse/YARN-7200
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler-load-simulator
> Reporter: Grant Sohn
> Assignee: Agshin Kazimli
> Priority: Minor
> Labels: newbie, newbie++
> Attachments: YARN-7200-branch-trunk.patch, realtimetrack.json,
> snemeth-testing-20201113.zip
>
>
> File
> hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerMetrics.java
> shows:
> {noformat}
> void tearDown() throws Exception {
> if (metricsLogBW != null) {
> metricsLogBW.write("]");
> metricsLogBW.close();
> }
> ....
> {noformat}
> So the exit logic is flawed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
