[jira] [Commented] (YARN-4183) Enabling generic application history forces every job to get a timeline service delegation token
[ https://issues.apache.org/jira/browse/YARN-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908048#comment-14908048 ] Mit Desai commented on YARN-4183: - Here is the scenario. We want the Yarn application to not use the timeline server during execution but use the application history server for the logs. This will not be possible with the current implementation. It is either both or none. If we check for application history enabled, it indirectly tells that timeline service is enable. Because history server will not be enabled without enabling the timeline server. This way, the system metrics publisher can publish events to the history server even if the applications do not use the timeline server for execution. > Enabling generic application history forces every job to get a timeline > service delegation token > > > Key: YARN-4183 > URL: https://issues.apache.org/jira/browse/YARN-4183 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-4183.1.patch > > > When enabling just the Generic History Server and not the timeline server, > the system metrics publisher will not publish the events to the timeline > store as it checks if the timeline server and system metrics publisher are > enabled before creating a timeline client. > To make it work, if the timeline service flag is turned on, it will force > every yarn application to get a delegation token. > Instead of checking if timeline service is enabled, we should be checking if > application history server is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3679) Add documentation for timeline server filter ordering
[ https://issues.apache.org/jira/browse/YARN-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907987#comment-14907987 ] Mit Desai commented on YARN-3679: - [~xgong], thanks for looking. YARN-3624 only changes the timeline server. But that includes the authentication part as well. Filter initialization is done during the authentication. So I have made the change in HttpAuthentication. > Add documentation for timeline server filter ordering > - > > Key: YARN-3679 > URL: https://issues.apache.org/jira/browse/YARN-3679 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3679.patch > > > Currently the auth filter is before static user filter by default. After > YARN-3624, the filter order is no longer reversed. So the pseudo auth's > allowing anonymous config is useless with both filters loaded in the new > order, because static user will be created before presenting it to auth > filter. The user can remove static user filter from the config to get > anonymous user work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3976) Catch ApplicationNotFoundException instead of parent YarnException in YarnClient and AppReportFetcher
[
https://issues.apache.org/jira/browse/YARN-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14904940#comment-14904940
]
Mit Desai commented on YARN-3976:
-
[~jlowe], Sorry did not see your comment to move this jira out of 2.7.2 until
now.
> Catch ApplicationNotFoundException instead of parent YarnException in
> YarnClient and AppReportFetcher
> -
>
> Key: YARN-3976
> URL: https://issues.apache.org/jira/browse/YARN-3976
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Mit Desai
>Assignee: Mit Desai
>Priority: Trivial
> Fix For: 2.7.2
>
>
> It's is better to catch the ApplicationNotFoundException rather than the
> parent YarnException and rethrow it when it's not ApplicationNotFoundExcepton
> {noformat}
> catch (YarnException e) {
> if (!historyServiceEnabled) {
> // Just throw it as usual if historyService is not enabled.
> throw e;
> }
> // Even if history-service is enabled, treat all exceptions still the
> same
> // except the following
> if (!(e.getClass() == ApplicationNotFoundException.class)) {
> throw e;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (YARN-3976) Catch ApplicationNotFoundException instead of parent YarnException in YarnClient and AppReportFetcher
[
https://issues.apache.org/jira/browse/YARN-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mit Desai resolved YARN-3976.
-
Resolution: Fixed
Fix Version/s: 2.7.2
Made the change along with YARN-3975.
Resolving this jira as YARN-3975 is committed.
> Catch ApplicationNotFoundException instead of parent YarnException in
> YarnClient and AppReportFetcher
> -
>
> Key: YARN-3976
> URL: https://issues.apache.org/jira/browse/YARN-3976
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Mit Desai
>Assignee: Mit Desai
>Priority: Trivial
> Fix For: 2.7.2
>
>
> It's is better to catch the ApplicationNotFoundException rather than the
> parent YarnException and rethrow it when it's not ApplicationNotFoundExcepton
> {noformat}
> catch (YarnException e) {
> if (!historyServiceEnabled) {
> // Just throw it as usual if historyService is not enabled.
> throw e;
> }
> // Even if history-service is enabled, treat all exceptions still the
> same
> // except the following
> if (!(e.getClass() == ApplicationNotFoundException.class)) {
> throw e;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[
https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14904767#comment-14904767
]
Mit Desai commented on YARN-3975:
-
Moreover this is the exact same change form the one that was uploaded
previously and got a +1.
Only difference between the two trunk patches is:
{{@Test//(timeout=5000)}} and {{@Test(timeout=5000)}}
> WebAppProxyServlet should not redirect to RM page if AHS is enabled
> ---
>
> Key: YARN-3975
> URL: https://issues.apache.org/jira/browse/YARN-3975
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch,
> YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch,
> YARN-3975.8.patch, YARN-3975.9.b2.7.patch, YARN-3975.9.patch,
> YARN-3975.9.patch
>
>
> WebAppProxyServlet should be updated to handle the case when the appreport
> doesn't have a tracking URL and the Application History Server is eanbled.
> As we would have already tried the RM and got the
> ApplicationNotFoundException we should not direct the user to the RM app page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-4202) TestYarnClient#testReservationAPIs fails intermittently
Mit Desai created YARN-4202:
---
Summary: TestYarnClient#testReservationAPIs fails intermittently
Key: YARN-4202
URL: https://issues.apache.org/jira/browse/YARN-4202
Project: Hadoop YARN
Issue Type: Bug
Reporter: Mit Desai
Fix For: 3.0.0
Found this failure while looking at the Pre-run on one of my Jiras.
{noformat}
org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
The planning algorithm could not find a valid allocation for your request
at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1149)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitReservation(ApplicationClientProtocolPBServiceImpl.java:428)
at
org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:465)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2230)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2224)
Caused by:
org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
The planning algorithm could not find a valid allocation for your request
at
org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.allocateUser(PlanningAlgorithm.java:69)
at
org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.createReservation(PlanningAlgorithm.java:140)
at
org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.TryManyReservationAgents.createReservation(TryManyReservationAgents.java:55)
at
org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.AlignedPlannerWithGreedy.createReservation(AlignedPlannerWithGreedy.java:84)
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1132)
... 10 more
{noformat}
TestReport Link:
https://builds.apache.org/job/PreCommit-YARN-Build/9243/testReport/
When I ran this on my local box branch-2, it succeeds.
{noformat}
---
T E S T S
---
Running org.apache.hadoop.yarn.client.api.impl.TestYarnClient
Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.999 sec -
in org.apache.hadoop.yarn.client.api.impl.TestYarnClient
Results :
Tests run: 21, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 52.029 s
[INFO] Finished at: 2015-09-23T11:25:04-06:00
[INFO] Final Memory: 31M/391M
[INFO]
{noformat}
Haven't tried if it is a problem in branch-2.7 or not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[
https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14904762#comment-14904762
]
Mit Desai commented on YARN-3975:
-
Findbugs not broken due to the patch.
This is from the console output of this run.
{noformat}
Running findbugs in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
/home/jenkins/tools/maven/latest/bin/mvn clean test findbugs:findbugs
-DskipTests -DhadoopPatchProcess >
/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/trunkFindBugsOutputhadoop-yarn-client.txt
2>&1
Exception in thread "main" java.io.FileNotFoundException:
/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/trunkFindbugsWarningshadoop-yarn-client.xml
(No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:146)
at
edu.umd.cs.findbugs.SortedBugCollection.progessMonitoredInputStream(SortedBugCollection.java:1231)
at
edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:308)
at
edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:295)
at edu.umd.cs.findbugs.workflow.Filter.main(Filter.java:712)
{noformat}
Found a similar issue in YARN-4075.
Console Output: https://builds.apache.org/job/PreCommit-YARN-Build/9241/console
TestYarnClient passes when I run it on my local box.
Filed HADOOP-12435 and YARN-4202 for investigating the issues.
> WebAppProxyServlet should not redirect to RM page if AHS is enabled
> ---
>
> Key: YARN-3975
> URL: https://issues.apache.org/jira/browse/YARN-3975
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch,
> YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch,
> YARN-3975.8.patch, YARN-3975.9.b2.7.patch, YARN-3975.9.patch,
> YARN-3975.9.patch
>
>
> WebAppProxyServlet should be updated to handle the case when the appreport
> doesn't have a tracking URL and the Application History Server is eanbled.
> As we would have already tried the RM and got the
> ApplicationNotFoundException we should not direct the user to the RM app page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3975: Attachment: YARN-3975.9.b2.7.patch > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch, > YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch, > YARN-3975.8.patch, YARN-3975.9.b2.7.patch, YARN-3975.9.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3975: Attachment: YARN-3975.9.patch Attached branch-2.7 patch. Also attached b2 patch agian. I realized, I left a debug statement in the previous one. > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch, > YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch, > YARN-3975.8.patch, YARN-3975.9.b2.7.patch, YARN-3975.9.patch, > YARN-3975.9.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3975: Attachment: YARN-3975.9.patch Somehow attached a wrong version of patch previously. Attached the patch with checkstyle fixed. > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch, > YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch, > YARN-3975.8.patch, YARN-3975.9.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3975: Attachment: YARN-3975.8.patch > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch, > YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch, > YARN-3975.8.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4183) Enabling generic application history forces every job to get a timeline service delegation token
[ https://issues.apache.org/jira/browse/YARN-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-4183: Attachment: YARN-4183.1.patch Attaching the patch > Enabling generic application history forces every job to get a timeline > service delegation token > > > Key: YARN-4183 > URL: https://issues.apache.org/jira/browse/YARN-4183 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-4183.1.patch > > > When enabling just the Generic History Server and not the timeline server, > the system metrics publisher will not publish the events to the timeline > store as it checks if the timeline server and system metrics publisher are > enabled before creating a timeline client. > To make it work, if the timeline service flag is turned on, it will force > every yarn application to get a delegation token. > Instead of checking if timeline service is enabled, we should be checking if > application history server is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4183) Enabling generic application history forces every job to get a timeline service delegation token
Mit Desai created YARN-4183: --- Summary: Enabling generic application history forces every job to get a timeline service delegation token Key: YARN-4183 URL: https://issues.apache.org/jira/browse/YARN-4183 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.7.1 Reporter: Mit Desai Assignee: Mit Desai When enabling just the Generic History Server and not the timeline server, the system metrics publisher will not publish the events to the timeline store as it checks if the timeline server and system metrics publisher are enabled before creating a timeline client. To make it work, if the timeline service flag is turned on, it will force every yarn application to get a delegation token. Instead of checking if timeline service is enabled, we should be checking if application history server is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3975: Attachment: YARN-3975.7.patch [~jlowe] Thanks for taking a look. I have updated the patch and incorporated your comments. Can you please have another look? > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch, > YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch, YARN-3975.7.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3975: Attachment: YARN-3975.6.patch Fixed Checkstyle issues > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch, > YARN-3975.4.patch, YARN-3975.5.patch, YARN-3975.6.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3975: Attachment: YARN-3975.5.patch Thanks for the review [~jlowe]. I have modified the patch based on your comments. > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch, > YARN-3975.4.patch, YARN-3975.5.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[
https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14680943#comment-14680943
]
Mit Desai commented on YARN-3975:
-
The test does not fail for me.
{noformat}
localhost:hadoop-yarn-common mitdesai$ git status
On branch trunk
Your branch is up-to-date with 'origin/trunk'.
Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)
modified:
../hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java
modified:
../hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
modified:
src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java
modified:
../hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java
modified:
../hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
modified:
../hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java
no changes added to commit (use "git add" and/or "git commit -a")
localhost:hadoop-yarn-common mitdesai$ mvn clean test -Dtest=TestRackResolver
[INFO] Scanning for projects...
[INFO]
[INFO] Using the builder
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder
with a thread count of 1
[INFO]
[INFO]
[INFO] Building Apache Hadoop YARN Common 3.0.0-SNAPSHOT
[INFO]
... skipping ...
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.yarn.util.TestRackResolver
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.412 sec - in
org.apache.hadoop.yarn.util.TestRackResolver
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 15.523 s
[INFO] Finished at: 2015-08-10T18:19:15-06:00
[INFO] Final Memory: 38M/577M
[INFO]
{noformat}
> WebAppProxyServlet should not redirect to RM page if AHS is enabled
> ---
>
> Key: YARN-3975
> URL: https://issues.apache.org/jira/browse/YARN-3975
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch,
> YARN-3975.4.patch
>
>
> WebAppProxyServlet should be updated to handle the case when the appreport
> doesn't have a tracking URL and the Application History Server is eanbled.
> As we would have already tried the RM and got the
> ApplicationNotFoundException we should not direct the user to the RM app page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3975: Attachment: YARN-3975.4.patch Updated patch. Also fixed YarnClient and WebAppProxyServlet catching YarnException and throwing the exception if it is not ApplicationNotFoundException > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch, > YARN-3975.4.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3975: Attachment: YARN-3975.3.patch Refined the patch. The test failure is not related to the patch. I tested it on my local machine but could not reproduce it > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3975.2.b2.patch, YARN-3975.3.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
[ https://issues.apache.org/jira/browse/YARN-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3975: Attachment: YARN-3975.2.b2.patch Attaching patch > WebAppProxyServlet should not redirect to RM page if AHS is enabled > --- > > Key: YARN-3975 > URL: https://issues.apache.org/jira/browse/YARN-3975 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3975.2.b2.patch > > > WebAppProxyServlet should be updated to handle the case when the appreport > doesn't have a tracking URL and the Application History Server is eanbled. > As we would have already tried the RM and got the > ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3624) ApplicationHistoryServer reverses the order of the filters it gets
[ https://issues.apache.org/jira/browse/YARN-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644361#comment-14644361 ] Mit Desai commented on YARN-3624: - [~zjshen] / [~jeagles] did you guys find some time to take a look at this or YARN-3679? > ApplicationHistoryServer reverses the order of the filters it gets > -- > > Key: YARN-3624 > URL: https://issues.apache.org/jira/browse/YARN-3624 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3624.patch > > > AppliactionHistoryServer should not alter the order in which it gets the > filter chain. Additional filters should be added at the end of the chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3967) Fetch the application report from the AHS if the RM does not know about it
[ https://issues.apache.org/jira/browse/YARN-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14641053#comment-14641053 ] Mit Desai commented on YARN-3967: - Filed YARN-3976 > Fetch the application report from the AHS if the RM does not know about it > -- > > Key: YARN-3967 > URL: https://issues.apache.org/jira/browse/YARN-3967 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 2.7.2 > > Attachments: YARN-3967.1.patch, YARN-3967.2.patch, YARN-3967.3.patch > > > If the application history service has been enabled and RM has forgotten > anout an application, try and fetch the app report form the AHS. > On larger clusters, the RM can forget about the applications in about 30 > minutes. The proxy url generated during the job submission will try to fetch > the app report from the RM and will fail to get anything from there. If the > app is not found in the RM, we will need to get the application report from > the Application History Server (if it is enabled) to see if we can get any > information on that application before throwing an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3976) Catch ApplicationNotFoundException instead of parent YarnException in YarnClient and AppReportFetcher
Mit Desai created YARN-3976:
---
Summary: Catch ApplicationNotFoundException instead of parent
YarnException in YarnClient and AppReportFetcher
Key: YARN-3976
URL: https://issues.apache.org/jira/browse/YARN-3976
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.7.1
Reporter: Mit Desai
Assignee: Mit Desai
It's is better to catch the ApplicationNotFoundException rather than the parent
YarnException and rethrow it when it's not ApplicationNotFoundExcepton
{noformat}
catch (YarnException e) {
if (!historyServiceEnabled) {
// Just throw it as usual if historyService is not enabled.
throw e;
}
// Even if history-service is enabled, treat all exceptions still the same
// except the following
if (!(e.getClass() == ApplicationNotFoundException.class)) {
throw e;
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3967) Fetch the application report from the AHS if the RM does not know about it
[ https://issues.apache.org/jira/browse/YARN-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14641046#comment-14641046 ] Mit Desai commented on YARN-3967: - Agreed. We should make the change in at both places than. I'll file a Jira for that as well. > Fetch the application report from the AHS if the RM does not know about it > -- > > Key: YARN-3967 > URL: https://issues.apache.org/jira/browse/YARN-3967 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 2.7.2 > > Attachments: YARN-3967.1.patch, YARN-3967.2.patch, YARN-3967.3.patch > > > If the application history service has been enabled and RM has forgotten > anout an application, try and fetch the app report form the AHS. > On larger clusters, the RM can forget about the applications in about 30 > minutes. The proxy url generated during the job submission will try to fetch > the app report from the RM and will fail to get anything from there. If the > app is not found in the RM, we will need to get the application report from > the Application History Server (if it is enabled) to see if we can get any > information on that application before throwing an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3967) Fetch the application report from the AHS if the RM does not know about it
[ https://issues.apache.org/jira/browse/YARN-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14640913#comment-14640913 ] Mit Desai commented on YARN-3967: - * YARN-3975 > Fetch the application report from the AHS if the RM does not know about it > -- > > Key: YARN-3967 > URL: https://issues.apache.org/jira/browse/YARN-3967 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 2.7.2 > > Attachments: YARN-3967.1.patch, YARN-3967.2.patch, YARN-3967.3.patch > > > If the application history service has been enabled and RM has forgotten > anout an application, try and fetch the app report form the AHS. > On larger clusters, the RM can forget about the applications in about 30 > minutes. The proxy url generated during the job submission will try to fetch > the app report from the RM and will fail to get anything from there. If the > app is not found in the RM, we will need to get the application report from > the Application History Server (if it is enabled) to see if we can get any > information on that application before throwing an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3975) WebAppProxyServlet should not redirect to RM page if AHS is enabled
Mit Desai created YARN-3975: --- Summary: WebAppProxyServlet should not redirect to RM page if AHS is enabled Key: YARN-3975 URL: https://issues.apache.org/jira/browse/YARN-3975 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.7.1 Reporter: Mit Desai Assignee: Mit Desai WebAppProxyServlet should be updated to handle the case when the appreport doesn't have a tracking URL and the Application History Server is eanbled. As we would have already tried the RM and got the ApplicationNotFoundException we should not direct the user to the RM app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3967) Fetch the application report from the AHS if the RM does not know about it
[ https://issues.apache.org/jira/browse/YARN-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14640912#comment-14640912 ] Mit Desai commented on YARN-3967: - Created YARN-2975 > Fetch the application report from the AHS if the RM does not know about it > -- > > Key: YARN-3967 > URL: https://issues.apache.org/jira/browse/YARN-3967 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 2.7.2 > > Attachments: YARN-3967.1.patch, YARN-3967.2.patch, YARN-3967.3.patch > > > If the application history service has been enabled and RM has forgotten > anout an application, try and fetch the app report form the AHS. > On larger clusters, the RM can forget about the applications in about 30 > minutes. The proxy url generated during the job submission will try to fetch > the app report from the RM and will fail to get anything from there. If the > app is not found in the RM, we will need to get the application report from > the Application History Server (if it is enabled) to see if we can get any > information on that application before throwing an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3967) Fetch the application report from the AHS if the RM does not know about it
[ https://issues.apache.org/jira/browse/YARN-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14640849#comment-14640849 ] Mit Desai commented on YARN-3967: - bq. I'm confused why we are not simply adding a separate explicit clause to catch ApplicationNotFoundException [~jlowe] This was to keep the implementation consistent with what the YarnClient does. If we want to make the change, I would suggest changing both YarnClient and AppReportFetcher to behave in a similar manner. bq. This code also needs to be updated to handle the case when the appreport doesn't have a tracking URL and we got the report from the history server, as we will direct the user to the RM app page. We already know the RM app page doesn't exist due to the ApplicationNotFoundException earlier: Nice catch. This is a rare case that should not happen in practice. I'll file another Jira to fix that issue. > Fetch the application report from the AHS if the RM does not know about it > -- > > Key: YARN-3967 > URL: https://issues.apache.org/jira/browse/YARN-3967 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 2.7.2 > > Attachments: YARN-3967.1.patch, YARN-3967.2.patch, YARN-3967.3.patch > > > If the application history service has been enabled and RM has forgotten > anout an application, try and fetch the app report form the AHS. > On larger clusters, the RM can forget about the applications in about 30 > minutes. The proxy url generated during the job submission will try to fetch > the app report from the RM and will fail to get anything from there. If the > app is not found in the RM, we will need to get the application report from > the Application History Server (if it is enabled) to see if we can get any > information on that application before throwing an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3967) Fetch the application report from the AHS if the RM does not know about it
[ https://issues.apache.org/jira/browse/YARN-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3967: Attachment: YARN-3967.3.patch Refined the patch [~xgong], I actually tried doing that before. But historyManager is a final member and cannot be initialiez outside. So it is not possible to make that change. There are 2 constructors for this class so initialization will have to be in 2 places. > Fetch the application report from the AHS if the RM does not know about it > -- > > Key: YARN-3967 > URL: https://issues.apache.org/jira/browse/YARN-3967 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3967.1.patch, YARN-3967.2.patch, YARN-3967.3.patch > > > If the application history service has been enabled and RM has forgotten > anout an application, try and fetch the app report form the AHS. > On larger clusters, the RM can forget about the applications in about 30 > minutes. The proxy url generated during the job submission will try to fetch > the app report from the RM and will fail to get anything from there. If the > app is not found in the RM, we will need to get the application report from > the Application History Server (if it is enabled) to see if we can get any > information on that application before throwing an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3967) Fetch the application report from the AHS if the RM does not know about it
[ https://issues.apache.org/jira/browse/YARN-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3967: Attachment: YARN-3967.2.patch Fixed checkstyle and whitespaces > Fetch the application report from the AHS if the RM does not know about it > -- > > Key: YARN-3967 > URL: https://issues.apache.org/jira/browse/YARN-3967 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3967.1.patch, YARN-3967.2.patch > > > If the application history service has been enabled and RM has forgotten > anout an application, try and fetch the app report form the AHS. > On larger clusters, the RM can forget about the applications in about 30 > minutes. The proxy url generated during the job submission will try to fetch > the app report from the RM and will fail to get anything from there. If the > app is not found in the RM, we will need to get the application report from > the Application History Server (if it is enabled) to see if we can get any > information on that application before throwing an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3967) Fetch the application report from the AHS if the RM does not know about it
[ https://issues.apache.org/jira/browse/YARN-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3967: Attachment: YARN-3967.1.patch Attaching the patch > Fetch the application report from the AHS if the RM does not know about it > -- > > Key: YARN-3967 > URL: https://issues.apache.org/jira/browse/YARN-3967 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3967.1.patch > > > If the application history service has been enabled and RM has forgotten > anout an application, try and fetch the app report form the AHS. > On larger clusters, the RM can forget about the applications in about 30 > minutes. The proxy url generated during the job submission will try to fetch > the app report from the RM and will fail to get anything from there. If the > app is not found in the RM, we will need to get the application report from > the Application History Server (if it is enabled) to see if we can get any > information on that application before throwing an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-3967) Fetch the application report from the AHS if the RM does not know about it
[ https://issues.apache.org/jira/browse/YARN-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai reassigned YARN-3967: --- Assignee: Mit Desai > Fetch the application report from the AHS if the RM does not know about it > -- > > Key: YARN-3967 > URL: https://issues.apache.org/jira/browse/YARN-3967 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Mit Desai > > If the application history service has been enabled and RM has forgotten > anout an application, try and fetch the app report form the AHS. > On larger clusters, the RM can forget about the applications in about 30 > minutes. The proxy url generated during the job submission will try to fetch > the app report from the RM and will fail to get anything from there. If the > app is not found in the RM, we will need to get the application report from > the Application History Server (if it is enabled) to see if we can get any > information on that application before throwing an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3967) Fetch the application report from the AHS if the RM does not know about it
Mit Desai created YARN-3967: --- Summary: Fetch the application report from the AHS if the RM does not know about it Key: YARN-3967 URL: https://issues.apache.org/jira/browse/YARN-3967 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.7.1 Reporter: Mit Desai If the application history service has been enabled and RM has forgotten anout an application, try and fetch the app report form the AHS. On larger clusters, the RM can forget about the applications in about 30 minutes. The proxy url generated during the job submission will try to fetch the app report from the RM and will fail to get anything from there. If the app is not found in the RM, we will need to get the application report from the Application History Server (if it is enabled) to see if we can get any information on that application before throwing an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1965) Interrupted exception when closing YarnClient
[
https://issues.apache.org/jira/browse/YARN-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605955#comment-14605955
]
Mit Desai commented on YARN-1965:
-
latest patch looks good.
+1 (non-binding)
> Interrupted exception when closing YarnClient
> -
>
> Key: YARN-1965
> URL: https://issues.apache.org/jira/browse/YARN-1965
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Affects Versions: 2.3.0
>Reporter: Oleg Zhurakousky
>Assignee: Kuhu Shukla
>Priority: Minor
> Labels: newbie
> Attachments: YARN-1965-v2.patch, YARN-1965-v3.patch, YARN-1965.patch
>
>
> Its more of a nuisance then a bug, but nevertheless
> {code}
> 16:16:48,709 ERROR pool-1-thread-1 ipc.Client:195 - Interrupted while waiting
> for clientExecutorto stop
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2072)
> at
> java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1468)
> at
> org.apache.hadoop.ipc.Client$ClientExecutorServiceFactory.unrefAndCleanup(Client.java:191)
> at org.apache.hadoop.ipc.Client.stop(Client.java:1235)
> at org.apache.hadoop.ipc.ClientCache.stopClient(ClientCache.java:100)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.close(ProtobufRpcEngine.java:251)
> at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:626)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.close(ApplicationClientProtocolPBClientImpl.java:112)
> at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:621)
> at
> org.apache.hadoop.io.retry.DefaultFailoverProxyProvider.close(DefaultFailoverProxyProvider.java:57)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.close(RetryInvocationHandler.java:206)
> at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:626)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStop(YarnClientImpl.java:124)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> . . .
> {code}
> It happens sporadically when stopping YarnClient.
> Looking at the code in Client's 'unrefAndCleanup' its not immediately obvious
> why and who throws the interrupt but in any event it should not be logged as
> ERROR. Probably a WARN with no stack trace.
> Also, for consistency and correctness you may want to Interrupt current
> thread as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1965) Interrupted exception when closing YarnClient
[
https://issues.apache.org/jira/browse/YARN-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14603103#comment-14603103
]
Mit Desai commented on YARN-1965:
-
Overall patch Looks good. Few minor nits.
* There should be a space between () and { here
{{public static final ExecutorService getClientExecutor(){}}
* In testStandAloneClient(), we need spaces near the brackets. Change
{{}finally{}} to {{} finally {}}
* In testConnectionIdleTimeouts(), we need space near the brackets. Change
{{}finally{}} to {{} finally {}}
* testInterrupted needs to be indented.
* In doErrorTest, testRTEDuringConnectionSetup stopping the client before the
server makes more sense. Swap the stop calls in finally block
* In testSocketFactoryException,testIpcConnectTimeout {{client.stop()}} should
be within finally block
* Is there a need to move {{Client client = new Client(LongWritable.class,
conf, spyFactory);}} in testRTEDuringConnectionSetup?
> Interrupted exception when closing YarnClient
> -
>
> Key: YARN-1965
> URL: https://issues.apache.org/jira/browse/YARN-1965
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Affects Versions: 2.3.0
>Reporter: Oleg Zhurakousky
>Assignee: Kuhu Shukla
>Priority: Minor
> Labels: newbie
> Attachments: YARN-1965-v2.patch, YARN-1965.patch
>
>
> Its more of a nuisance then a bug, but nevertheless
> {code}
> 16:16:48,709 ERROR pool-1-thread-1 ipc.Client:195 - Interrupted while waiting
> for clientExecutorto stop
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2072)
> at
> java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1468)
> at
> org.apache.hadoop.ipc.Client$ClientExecutorServiceFactory.unrefAndCleanup(Client.java:191)
> at org.apache.hadoop.ipc.Client.stop(Client.java:1235)
> at org.apache.hadoop.ipc.ClientCache.stopClient(ClientCache.java:100)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.close(ProtobufRpcEngine.java:251)
> at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:626)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.close(ApplicationClientProtocolPBClientImpl.java:112)
> at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:621)
> at
> org.apache.hadoop.io.retry.DefaultFailoverProxyProvider.close(DefaultFailoverProxyProvider.java:57)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.close(RetryInvocationHandler.java:206)
> at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:626)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStop(YarnClientImpl.java:124)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> . . .
> {code}
> It happens sporadically when stopping YarnClient.
> Looking at the code in Client's 'unrefAndCleanup' its not immediately obvious
> why and who throws the interrupt but in any event it should not be logged as
> ERROR. Probably a WARN with no stack trace.
> Also, for consistency and correctness you may want to Interrupt current
> thread as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1965) Interrupted exception when closing YarnClient
[
https://issues.apache.org/jira/browse/YARN-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14601701#comment-14601701
]
Mit Desai commented on YARN-1965:
-
Thanks for the patch [~kshukla]. I will review it shortly.
> Interrupted exception when closing YarnClient
> -
>
> Key: YARN-1965
> URL: https://issues.apache.org/jira/browse/YARN-1965
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Affects Versions: 2.3.0
>Reporter: Oleg Zhurakousky
>Assignee: Kuhu Shukla
>Priority: Minor
> Labels: newbie
> Attachments: YARN-1965-v2.patch, YARN-1965.patch
>
>
> Its more of a nuisance then a bug, but nevertheless
> {code}
> 16:16:48,709 ERROR pool-1-thread-1 ipc.Client:195 - Interrupted while waiting
> for clientExecutorto stop
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2072)
> at
> java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1468)
> at
> org.apache.hadoop.ipc.Client$ClientExecutorServiceFactory.unrefAndCleanup(Client.java:191)
> at org.apache.hadoop.ipc.Client.stop(Client.java:1235)
> at org.apache.hadoop.ipc.ClientCache.stopClient(ClientCache.java:100)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.close(ProtobufRpcEngine.java:251)
> at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:626)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.close(ApplicationClientProtocolPBClientImpl.java:112)
> at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:621)
> at
> org.apache.hadoop.io.retry.DefaultFailoverProxyProvider.close(DefaultFailoverProxyProvider.java:57)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.close(RetryInvocationHandler.java:206)
> at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:626)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStop(YarnClientImpl.java:124)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> . . .
> {code}
> It happens sporadically when stopping YarnClient.
> Looking at the code in Client's 'unrefAndCleanup' its not immediately obvious
> why and who throws the interrupt but in any event it should not be logged as
> ERROR. Probably a WARN with no stack trace.
> Also, for consistency and correctness you may want to Interrupt current
> thread as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566294#comment-14566294 ] Mit Desai commented on YARN-2900: - looks good to me. Thanks [~zjshen] for your looking into it. > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Mit Desai > Attachments: YARN-2900-b2-2.patch, YARN-2900-b2.patch, > YARN-2900.20150529.patch, YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565106#comment-14565106 ] Mit Desai commented on YARN-2900: - Thanks! I am working on it currently and should have some update in couple hours > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Mit Desai > Attachments: YARN-2900-b2-2.patch, YARN-2900-b2.patch, > YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2900: Attachment: YARN-2900-b2-2.patch The latest patch does not apply anymore. Attaching it again so it applies cleanly. [~zjshen], sorry I could not get to this soon. > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Zhijie Shen > Attachments: YARN-2900-b2-2.patch, YARN-2900-b2.patch, > YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3679) Add documentation for timeline server filter ordering
[ https://issues.apache.org/jira/browse/YARN-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560959#comment-14560959 ] Mit Desai commented on YARN-3679: - [~zjshen]/[~jeagles] did you guys get a chance to take a look at this? > Add documentation for timeline server filter ordering > - > > Key: YARN-3679 > URL: https://issues.apache.org/jira/browse/YARN-3679 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3679.patch > > > Currently the auth filter is before static user filter by default. After > YARN-3624, the filter order is no longer reversed. So the pseudo auth's > allowing anonymous config is useless with both filters loaded in the new > order, because static user will be created before presenting it to auth > filter. The user can remove static user filter from the config to get > anonymous user work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3679) Add documentation for timeline server filter ordering
[ https://issues.apache.org/jira/browse/YARN-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3679: Attachment: YARN-3679.patch [~jeagles], [~zjshen], can you take a look on the patch? > Add documentation for timeline server filter ordering > - > > Key: YARN-3679 > URL: https://issues.apache.org/jira/browse/YARN-3679 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3679.patch > > > Currently the auth filter is before static user filter by default. After > YARN-3624, the filter order is no longer reversed. So the pseudo auth's > allowing anonymous config is useless with both filters loaded in the new > order, because static user will be created before presenting it to auth > filter. The user can remove static user filter from the config to get > anonymous user work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3624) ApplicationHistoryServer reverses the order of the filters it gets
[ https://issues.apache.org/jira/browse/YARN-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14550479#comment-14550479 ] Mit Desai commented on YARN-3624: - Correction: YARN-3679 > ApplicationHistoryServer reverses the order of the filters it gets > -- > > Key: YARN-3624 > URL: https://issues.apache.org/jira/browse/YARN-3624 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3624.patch > > > AppliactionHistoryServer should not alter the order in which it gets the > filter chain. Additional filters should be added at the end of the chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-3679) Add documentation for timeline server filter ordering
[ https://issues.apache.org/jira/browse/YARN-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai reassigned YARN-3679: --- Assignee: Mit Desai > Add documentation for timeline server filter ordering > - > > Key: YARN-3679 > URL: https://issues.apache.org/jira/browse/YARN-3679 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Mit Desai >Assignee: Mit Desai > > Currently the auth filter is before static user filter by default. After > YARN-3624, the filter order is no longer reversed. So the pseudo auth's > allowing anonymous config is useless with both filters loaded in the new > order, because static user will be created before presenting it to auth > filter. The user can remove static user filter from the config to get > anonymous user work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3679) Add documentation for timeline server filter ordering
Mit Desai created YARN-3679: --- Summary: Add documentation for timeline server filter ordering Key: YARN-3679 URL: https://issues.apache.org/jira/browse/YARN-3679 Project: Hadoop YARN Issue Type: Bug Reporter: Mit Desai Currently the auth filter is before static user filter by default. After YARN-3624, the filter order is no longer reversed. So the pseudo auth's allowing anonymous config is useless with both filters loaded in the new order, because static user will be created before presenting it to auth filter. The user can remove static user filter from the config to get anonymous user work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3624) ApplicationHistoryServer reverses the order of the filters it gets
[ https://issues.apache.org/jira/browse/YARN-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14550447#comment-14550447 ] Mit Desai commented on YARN-3624: - Filed YARN-2679 > ApplicationHistoryServer reverses the order of the filters it gets > -- > > Key: YARN-3624 > URL: https://issues.apache.org/jira/browse/YARN-3624 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3624.patch > > > AppliactionHistoryServer should not alter the order in which it gets the > filter chain. Additional filters should be added at the end of the chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3624) ApplicationHistoryServer reverses the order of the filters it gets
[ https://issues.apache.org/jira/browse/YARN-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14545968#comment-14545968 ] Mit Desai commented on YARN-3624: - I can add that. Thanks for reviewing the patch. [~zjshen], [~jeagles] Should we make that change as a part of a separate JIRA in this one? I think it would be good to have a documentation JIRA separate. But I am OK either way. > ApplicationHistoryServer reverses the order of the filters it gets > -- > > Key: YARN-3624 > URL: https://issues.apache.org/jira/browse/YARN-3624 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3624.patch > > > AppliactionHistoryServer should not alter the order in which it gets the > filter chain. Additional filters should be added at the end of the chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14545959#comment-14545959 ] Mit Desai commented on YARN-2900: - I am yet to verify that. Will try to get to this soon > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Mit Desai > Attachments: YARN-2900-b2.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3624) ApplicationHistoryServer reverses the order of the filters it gets
[ https://issues.apache.org/jira/browse/YARN-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544185#comment-14544185 ] Mit Desai commented on YARN-3624: - [~jeagles] can you take a look on this? > ApplicationHistoryServer reverses the order of the filters it gets > -- > > Key: YARN-3624 > URL: https://issues.apache.org/jira/browse/YARN-3624 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3624.patch > > > AppliactionHistoryServer should not alter the order in which it gets the > filter chain. Additional filters should be added at the end of the chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3624) ApplicationHistoryServer reverses the order of the filters it gets
[ https://issues.apache.org/jira/browse/YARN-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3624: Target Version/s: 2.7.1 (was: 2.6.1) > ApplicationHistoryServer reverses the order of the filters it gets > -- > > Key: YARN-3624 > URL: https://issues.apache.org/jira/browse/YARN-3624 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3624.patch > > > AppliactionHistoryServer should not alter the order in which it gets the > filter chain. Additional filters should be added at the end of the chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539061#comment-14539061 ] Mit Desai commented on YARN-2900: - I was stuck in something else. I'll update on that by tomorrow > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Mit Desai > Attachments: YARN-2900-b2.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3624) ApplicationHistoryServer reverses the order of the filters it gets
[ https://issues.apache.org/jira/browse/YARN-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-3624: Attachment: YARN-3624.patch attaching the patch > ApplicationHistoryServer reverses the order of the filters it gets > -- > > Key: YARN-3624 > URL: https://issues.apache.org/jira/browse/YARN-3624 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-3624.patch > > > AppliactionHistoryServer should not alter the order in which it gets the > filter chain. Additional filters should be added at the end of the chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3624) ApplicationHistoryServer reverses the order of the filters it gets
Mit Desai created YARN-3624: --- Summary: ApplicationHistoryServer reverses the order of the filters it gets Key: YARN-3624 URL: https://issues.apache.org/jira/browse/YARN-3624 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 2.6.0 Reporter: Mit Desai Assignee: Mit Desai AppliactionHistoryServer should not alter the order in which it gets the filter chain. Additional filters should be added at the end of the chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2900: Attachment: YARN-2900-b2.patch > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Mit Desai > Attachments: YARN-2900-b2.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14534563#comment-14534563 ] Mit Desai commented on YARN-2900: - ApplicationHistoryClientService got changed recently. So this patch needs to be reworked. > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Mit Desai > Attachments: YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14534559#comment-14534559 ] Mit Desai commented on YARN-2900: - I'll update it shortly > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Mit Desai > Attachments: YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3573) MiniMRYarnCluster constructor that starts the timeline server using a boolean should be marked depricated
[
https://issues.apache.org/jira/browse/YARN-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529176#comment-14529176
]
Mit Desai commented on YARN-3573:
-
[~djp], YARN-2890 takes care of that concern.
[~brahmareddy], thanks for picking this Jira. I think there should be a similar
change to MiniYARNCluster.
{code}
public MiniYARNCluster(
String testName, int numResourceManagers, int numNodeManagers,
int numLocalDirs, int numLogDirs, boolean enableAHS) {
{code}
This method is ultimately called by MiniMRYarnCluster.
> MiniMRYarnCluster constructor that starts the timeline server using a boolean
> should be marked depricated
> -
>
> Key: YARN-3573
> URL: https://issues.apache.org/jira/browse/YARN-3573
> Project: Hadoop YARN
> Issue Type: Test
> Components: timelineserver
>Affects Versions: 2.6.0
>Reporter: Mit Desai
>Assignee: Brahma Reddy Battula
> Attachments: YARN-3573.patch
>
>
> {code}MiniMRYarnCluster(String testName, int noOfNMs, boolean enableAHS){code}
> starts the timeline server using *boolean enableAHS*. It is better to have
> the timelineserver started based on the config value.
> We should mark this constructor as deprecated to avoid its future use.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3573) MiniMRYarnCluster constructor that starts the timeline server using a boolean should be marked depricated
[
https://issues.apache.org/jira/browse/YARN-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mit Desai updated YARN-3573:
Assignee: (was: Mit Desai)
> MiniMRYarnCluster constructor that starts the timeline server using a boolean
> should be marked depricated
> -
>
> Key: YARN-3573
> URL: https://issues.apache.org/jira/browse/YARN-3573
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 2.6.0
>Reporter: Mit Desai
>
> {code}MiniMRYarnCluster(String testName, int noOfNMs, boolean enableAHS){code}
> starts the timeline server using *boolean enableAHS*. It is better to have
> the timelineserver started based on the config value.
> We should mark this constructor as deprecated to avoid its future use.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-3573) MiniMRYarnCluster constructor that starts the timeline server using a boolean should be marked depricated
[
https://issues.apache.org/jira/browse/YARN-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mit Desai reassigned YARN-3573:
---
Assignee: Mit Desai
> MiniMRYarnCluster constructor that starts the timeline server using a boolean
> should be marked depricated
> -
>
> Key: YARN-3573
> URL: https://issues.apache.org/jira/browse/YARN-3573
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 2.6.0
>Reporter: Mit Desai
>Assignee: Mit Desai
>
> {code}MiniMRYarnCluster(String testName, int noOfNMs, boolean enableAHS){code}
> starts the timeline server using *boolean enableAHS*. It is better to have
> the timelineserver started based on the config value.
> We should mark this constructor as deprecated to avoid its future use.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3573) MiniMRYarnCluster constructor that starts the timeline server using a boolean should be marked depricated
Mit Desai created YARN-3573:
---
Summary: MiniMRYarnCluster constructor that starts the timeline
server using a boolean should be marked depricated
Key: YARN-3573
URL: https://issues.apache.org/jira/browse/YARN-3573
Project: Hadoop YARN
Issue Type: Bug
Components: timelineserver
Affects Versions: 2.6.0
Reporter: Mit Desai
{code}MiniMRYarnCluster(String testName, int noOfNMs, boolean enableAHS){code}
starts the timeline server using *boolean enableAHS*. It is better to have the
timelineserver started based on the config value.
We should mark this constructor as deprecated to avoid its future use.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2365) TestAMRestart.testShouldNotCountFailureToMaxAttemptRetry fails on branch-2
[
https://issues.apache.org/jira/browse/YARN-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523897#comment-14523897
]
Mit Desai commented on YARN-2365:
-
Thanks for taking a look [~djp]
> TestAMRestart.testShouldNotCountFailureToMaxAttemptRetry fails on branch-2
> --
>
> Key: YARN-2365
> URL: https://issues.apache.org/jira/browse/YARN-2365
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: Mit Desai
>
> TestAMRestart#testShouldNotCountFailureToMaxAttemptRetry fails on branch-2
> with the following errror
> {noformat}
> Running
> org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 46.471 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
> testShouldNotCountFailureToMaxAttemptRetry(org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart)
> Time elapsed: 46.354 sec <<< FAILURE!
> java.lang.AssertionError: AppAttempt state is not correct (timedout)
> expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockAM.waitForState(MockAM.java:82)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:414)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.launchAM(MockRM.java:569)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.launchAndRegisterAM(MockRM.java:576)
> at
> org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart.testShouldNotCountFailureToMaxAttemptRetry(TestAMRestart.java:389)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3448) Add Rolling Time To Lives Level DB Plugin Capabilities
[
https://issues.apache.org/jira/browse/YARN-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517879#comment-14517879
]
Mit Desai commented on YARN-3448:
-
Thanks for the patch [~jeagles].
Few minor comments on the patch v13
Any reason why this is hard coded and not using
DEFAULT_TIMELINE_SERVICE_TTL_ENABLE
{code}
this.ttlEnabled = conf.getBoolean(
YarnConfiguration.TIMELINE_SERVICE_TTL_ENABLE, true);
{code}
The comment need to be fixed here. It should be 6 hours instead of 12
{code}
case QUARTER_DAILY: {
// round down to 12 hour interval
int hour = (cal.get(Calendar.HOUR) / 6) * 6;
{code}
Extra semicolon here
{code}
// seek to the first start time entry
iterator.seekToFirst();
;
{code}
Is just a log message enough when we catch the exception? 'db' will be null if
an exception is thrown
{code}
DB db = null;
try {
db = factory.open(new File(rollingDBPath.toUri().getPath()), options);
} catch (IOException ioe) {
LOG.warn("Failed to open rolling leveldb instance :"
+ new File(rollingDBPath.toUri().getPath()), ioe);
}
rollingdbs.put(dbStartTime, db);
String dbName = fdf.format(dbStartTime);
{code}
The TreeMap rollingdbs is mostly used for iteration. Would be faster to use an
ArrayList instead
{code}
/** Collection of all active rolling leveldb instances. */
private final TreeMap rollingdbs;
{code}
package-private would be more appropriate rather than marking private here
{code}
@VisibleForTesting
@Private
public synchronized long getStartTimeFor(DB db) {
long startTime = -1;
for (Map.Entry entry : rollingdbs.entrySet()) {
if (entry.getValue() == db) {
startTime = entry.getKey();
}
}
return startTime;
}
{code}
> Add Rolling Time To Lives Level DB Plugin Capabilities
> --
>
> Key: YARN-3448
> URL: https://issues.apache.org/jira/browse/YARN-3448
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: YARN-3448.1.patch, YARN-3448.10.patch,
> YARN-3448.12.patch, YARN-3448.13.patch, YARN-3448.2.patch, YARN-3448.3.patch,
> YARN-3448.4.patch, YARN-3448.5.patch, YARN-3448.7.patch, YARN-3448.8.patch,
> YARN-3448.9.patch
>
>
> For large applications, the majority of the time in LeveldbTimelineStore is
> spent deleting old entities record at a time. An exclusive write lock is held
> during the entire deletion phase which in practice can be hours. If we are to
> relax some of the consistency constraints, other performance enhancing
> techniques can be employed to maximize the throughput and minimize locking
> time.
> Split the 5 sections of the leveldb database (domain, owner, start time,
> entity, index) into 5 separate databases. This allows each database to
> maximize the read cache effectiveness based on the unique usage patterns of
> each database. With 5 separate databases each lookup is much faster. This can
> also help with I/O to have the entity and index databases on separate disks.
> Rolling DBs for entity and index DBs. 99.9% of the data are in these two
> sections 4:1 ration (index to entity) at least for tez. We replace DB record
> removal with file system removal if we create a rolling set of databases that
> age out and can be efficiently removed. To do this we must place a constraint
> to always place an entity's events into it's correct rolling db instance
> based on start time. This allows us to stitching the data back together while
> reading and artificial paging.
> Relax the synchronous writes constraints. If we are willing to accept losing
> some records that we not flushed in the operating system during a crash, we
> can use async writes that can be much faster.
> Prefer Sequential writes. sequential writes can be several times faster than
> random writes. Spend some small effort arranging the writes in such a way
> that will trend towards sequential write performance over random write
> performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3520) get rid of excessive stacktrace caused by expired cookie in timeline log
[
https://issues.apache.org/jira/browse/YARN-3520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505425#comment-14505425
]
Mit Desai commented on YARN-3520:
-
lgtm +1 (non-binding)
This change is related to logging so there is no need for tests.
> get rid of excessive stacktrace caused by expired cookie in timeline log
>
>
> Key: YARN-3520
> URL: https://issues.apache.org/jira/browse/YARN-3520
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
> Attachments: YARN-3520.patch
>
>
> {code}
> WARN sso.CookieValidatorHelpers: Cookie has expired by 25364187 msec
> WARN server.AuthenticationFilter: Authentication exception: Invalid Cookie
> 166 org.apache.hadoop.security.authentication.client.AuthenticationException:
> Invalid Bouncer Cookie
> 167 at
> KerberosAuthenticationHandler.bouncerAuthenticate(KerberosAuthenticationHandler.java:94)
> 168 at
> AuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:82)
> 169 at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:507)
> 170 at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> 171 at
> org.apache.hadoop.yarn.server.timeline.webapp.CrossOriginFilter.doFilter(CrossOriginFilter.java:95)
> 172 at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> 173 at
> org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
> 174 at GzipFilter.doFilter(GzipFilter.java:188)
> 175 at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> 176 at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1224)
> 177 at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> 178 at
> org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> 179 at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> 180 at
> org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> 181 at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> 182 at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> 183 at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> 184 at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> 185 at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> 186 at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> 187 at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> 188 at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> 189 at org.mortbay.jetty.Server.handle(Server.java:326)
> 190 at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> 191 at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> 192 at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> 193 at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> 194 at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> 195 at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> 196 at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> WARN sso.CookieValidatorHelpers: Cookie has expired by 25373197 msec
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3297) Changes for ResourceCalculatorProcessTree in YARN-3122 could be done in a more compatible manner
[ https://issues.apache.org/jira/browse/YARN-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503443#comment-14503443 ] Mit Desai commented on YARN-3297: - nice catch +1 (non-binding) > Changes for ResourceCalculatorProcessTree in YARN-3122 could be done in a > more compatible manner > > > Key: YARN-3297 > URL: https://issues.apache.org/jira/browse/YARN-3297 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: YARN-3297.1.patch > > > Related to YARN-3296, changes in YARN-3122 break any custom resource > monitoring plugin maintained outside of the YARN codebase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485303#comment-14485303 ] Mit Desai commented on YARN-2890: - [~hitesh] any comments on the latest patch? > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2890.1.patch, YARN-2890.2.patch, YARN-2890.3.patch, > YARN-2890.4.patch, YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, > YARN-2890.patch, YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14482488#comment-14482488 ] Mit Desai commented on YARN-2890: - These test failures are not related to the patch. These were also seen in MAPREDUCE-6293 which was not due to the patch. > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2890.1.patch, YARN-2890.2.patch, YARN-2890.3.patch, > YARN-2890.4.patch, YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, > YARN-2890.patch, YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2890: Attachment: YARN-2890.4.patch [~hitesh], Thanks for the comments. Attached updated patch. Created a new test file TestMiniYarnCluster that tests the the starting of timelineserver based on the configuration and enableAHS flag. > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2890.1.patch, YARN-2890.2.patch, YARN-2890.3.patch, > YARN-2890.4.patch, YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, > YARN-2890.patch, YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14393408#comment-14393408 ] Mit Desai commented on YARN-2890: - [~hitesh], did you had any comments on the patch? > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2890.1.patch, YARN-2890.2.patch, YARN-2890.3.patch, > YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, > YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387464#comment-14387464 ] Mit Desai commented on YARN-2890: - [~hitesh], the test failures were due to HADOOP-11754 > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2890.1.patch, YARN-2890.2.patch, YARN-2890.3.patch, > YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, > YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2890: Attachment: YARN-2890.3.patch Removed debug statements. > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2890.1.patch, YARN-2890.2.patch, YARN-2890.3.patch, > YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, > YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2890: Attachment: YARN-2890.2.patch Verified that the tests run fine on my box. Attaching another version of the patch where I fixed a silly mistake. > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2890.1.patch, YARN-2890.2.patch, YARN-2890.patch, > YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[
https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384275#comment-14384275
]
Mit Desai commented on YARN-2890:
-
The test {{testTimelineServiceStartInMiniCluster}} includes the following
scenarios.
1) Timeline service should not start if {{TIMELINE_SERVICE_ENABLED == false}}
and {{enableAHS}} is not set
2) Timeline service should not start if {{TIMELINE_SERVICE_ENABLED == false}}
and {{enableAHS == false}}
3) Timeline service should start if {{TIMELINE_SERVICE_ENABLED == true }}and
{{enableAHS == false}}
4) Timeline service should start if {{TIMELINE_SERVICE_ENABLED == false}} and
{{enableAHS == true}}
Following scenarios not included:
1) {{TIMELINE_SERVICE_ENABLED == true}} and {{enableAHS}} is not set:: This
case is already covered in the other tests in TestJobHistoryEventHandler
2) {{TIMELINE_SERVICE_ENABLED == true}} and {{enableAHS == true}}:: Timeline
service will start if either of this is true. This case will be a duplicate of
scenarios 3 and 4.
> MiniMRYarnCluster should turn on timeline service if configured to do so
>
>
> Key: YARN-2890
> URL: https://issues.apache.org/jira/browse/YARN-2890
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: YARN-2890.1.patch, YARN-2890.patch, YARN-2890.patch,
> YARN-2890.patch, YARN-2890.patch, YARN-2890.patch
>
>
> Currently the MiniMRYarnCluster does not consider the configuration value for
> enabling timeline service before starting. The MiniYarnCluster should only
> start the timeline service if it is configured to do so.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2890: Attachment: YARN-2890.1.patch [~hitesh], I have attached new patch addressing your comments. > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2890.1.patch, YARN-2890.patch, YARN-2890.patch, > YARN-2890.patch, YARN-2890.patch, YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14380894#comment-14380894 ] Mit Desai commented on YARN-2890: - I'm sorry for taking so long to get back [~hitesh]. bq. In the future, it would be good if your patches are versioned to avoid confusion. I will keep that in mind. But once a newer patch is submitted with the same name name, the older one gets grayed out. It was convenient for me. But if that creates confusion, I don't mind numbering the patches. bq. * testTimelineServiceStartInMiniCluster() - is there a reason why a job is run when timeline is enabled but not run when it is disabled? nice catch. That was by mistake. We do not need to test job run in this test. bq. * should be a job run be needed here in the first place given the name of the test? nope. we don't need a job run. I will update the patch to remove the that bq. * might be better to move the testing of job runs based on absence/presence of timeline to a separate test That can be done. But I don't think we are testing job runs here. Just the check for timeline service should be enough. bq. * testMRTimelineEventHandling, testMapreduceJobTimelineServiceEnabled, testMapreduceJobTimelineServiceEnabled: is there a need to change all of them? These tests are changed because checking the config value seems to be a better way to turn on/off the timelineserver than just passing a boolean value bq. * there does not seem to be a code path that tests timeline being enabled by passing the enableAHS value in the ctor if all these are changed. I would like to keep these tests as they are and add a new test that tests that code path. What do you think about that? [~hitesh], I will wait for your feedback before I post another patch for adding the test to test the enableAHS flag and remove the job run from the test that I added.. > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, > YARN-2890.patch, YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360938#comment-14360938 ] Mit Desai commented on YARN-2890: - bq. The above ctor was removed. If anyone is using MiniMRYARNCluster from 2.4.0 to test their jobs, this will break compatibility. My latest patch no longer removes this constructor bq. Why use a hardcoded false instead of the DEFAULT field from YarnConfiguration? Makes sense. Thanks. I will update the patch to use the default value which is set as false already. But I will wait for Zhijie's response before updating the patch. > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, > YARN-2890.patch, YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360704#comment-14360704 ] Mit Desai commented on YARN-2890: - Thats because not everything is using the timeline server. Turning it off by default will prevent users from accidentally using the timeline server if they do not intend to. Moreover if someone intends to use the timeline server, they are well aware and can turn the flag on. > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, > YARN-2890.patch, YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360415#comment-14360415 ] Mit Desai commented on YARN-2890: - [~hitesh] [~zjshen] Can you guys take a look? > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, > YARN-2890.patch, YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358798#comment-14358798 ] Mit Desai commented on YARN-2890: - Verified the test failures are not due to my patch. Some of them passes with my patch on my local machine. And some always fail for me. > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, > YARN-2890.patch, YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2890: Attachment: YARN-2890.patch Attaching the updated patch > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, > YARN-2890.patch, YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14355155#comment-14355155 ] Mit Desai commented on YARN-2890: - I was not aware of this Jira being reopened. I will take a look in a day or two. > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, > YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3238) Connection timeouts to nodemanagers are retried at multiple levels
[ https://issues.apache.org/jira/browse/YARN-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329998#comment-14329998 ] Mit Desai commented on YARN-3238: - +1 (non binding) Looks good to me > Connection timeouts to nodemanagers are retried at multiple levels > -- > > Key: YARN-3238 > URL: https://issues.apache.org/jira/browse/YARN-3238 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Attachments: YARN-3238.001.patch > > > The IPC layer will retry connection timeouts automatically (see Client.java), > but we are also retrying them with YARN's RetryPolicy put in place when the > NM proxy is created. This causes a two-level retry mechanism where the IPC > layer has already retried quite a few times (45 by default) for each YARN > RetryPolicy error that is retried. The end result is that NM clients can > wait a very, very long time for the connection to finally fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-2240) yarn logs can get corrupted if the aggregator does not have permissions to the log file it tries to read
[ https://issues.apache.org/jira/browse/YARN-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai resolved YARN-2240. - Resolution: Duplicate > yarn logs can get corrupted if the aggregator does not have permissions to > the log file it tries to read > > > Key: YARN-2240 > URL: https://issues.apache.org/jira/browse/YARN-2240 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Mit Desai > > When the log aggregator is aggregating the logs, it writes the file length > first. Then tries to open the log file and if it does not have permission to > do that, it ends up just writing an error message to the aggregated logs. > The mismatch between the file length and the actual length here makes the > aggregated logs corrupted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2240) yarn logs can get corrupted if the aggregator does not have permissions to the log file it tries to read
[ https://issues.apache.org/jira/browse/YARN-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315111#comment-14315111 ] Mit Desai commented on YARN-2240: - Thanks for pointing that out [~jlowe]. Closing this. > yarn logs can get corrupted if the aggregator does not have permissions to > the log file it tries to read > > > Key: YARN-2240 > URL: https://issues.apache.org/jira/browse/YARN-2240 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Mit Desai > > When the log aggregator is aggregating the logs, it writes the file length > first. Then tries to open the log file and if it does not have permission to > do that, it ends up just writing an error message to the aggregated logs. > The mismatch between the file length and the actual length here makes the > aggregated logs corrupted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2897) CrossOriginFilter needs more log statements
[ https://issues.apache.org/jira/browse/YARN-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2897: Attachment: YARN-2897.patch Updating the patch > CrossOriginFilter needs more log statements > --- > > Key: YARN-2897 > URL: https://issues.apache.org/jira/browse/YARN-2897 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2897.patch, YARN-2897.patch, YARN-2897.patch > > > CrossOriginFilter does not log as much to make debugging easier -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2897) CrossOriginFilter needs more log statements
[ https://issues.apache.org/jira/browse/YARN-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2897: Attachment: YARN-2897.patch Thanks for taking a look [~jeagles]. Attaching the modified patch > CrossOriginFilter needs more log statements > --- > > Key: YARN-2897 > URL: https://issues.apache.org/jira/browse/YARN-2897 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2897.patch, YARN-2897.patch > > > CrossOriginFilter does not log as much to make debugging easier -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2897) CrossOriginFilter needs more log statements
[ https://issues.apache.org/jira/browse/YARN-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2897: Description: CrossOriginFilter does not log as much to make debugging easier (was: CrossOriginFilter does not log as mcch to make debugging easier) > CrossOriginFilter needs more log statements > --- > > Key: YARN-2897 > URL: https://issues.apache.org/jira/browse/YARN-2897 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-2897.patch > > > CrossOriginFilter does not log as much to make debugging easier -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14240724#comment-14240724 ] Mit Desai commented on YARN-2900: - bq. I think you were still using the old FS history store. Yes, I did not make that config change. Verified after changing the history store that I can reproduce the problem too. > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Mit Desai > Attachments: YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[
https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14240170#comment-14240170
]
Mit Desai commented on YARN-2900:
-
[~zjshen], I did get an exception by following the steps you mentioned. But I
did not reproduce the scenario that you did.
This is the exception that I get in the timeline server logs.
{code}
2014-12-09 16:18:51,632 WARN [1920631011@qtp-530634372-8]
webapp.GenericExceptionHandler (GenericExceptionHandler.java:toResponse(98)) -
INTERNAL_SERVER_ERROR
javax.ws.rs.WebApplicationException: java.io.IOException: History file for
application application_1417818619773_0002 is not found
at
org.apache.hadoop.yarn.server.webapp.WebServices.rewrapAndThrowException(WebServices.java:458)
at
org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:227)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSWebServices.getApp(AHSWebServices.java:95)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:96)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:594)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:277)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:557)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1204)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212
[jira] [Updated] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2900: Attachment: YARN-2900.patch Refining patch. Missed to remove unused import > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Mit Desai > Attachments: YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2900: Attachment: YARN-2900.patch Attaching the patch that addresses the the NFE and indenting. I'll wait for your response on the IllegalStateException > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Mit Desai > Attachments: YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238255#comment-14238255 ] Mit Desai commented on YARN-2900: - [~zjshen], from the changes that the patch makes, only time that the NotFound is thrown is when there is no application|attempt|container that the client is asking for. I am not sure why the timelineserver throws some exception and we get a NotFound on the browser. Can you explain what was the test that you did here? > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Mit Desai > Attachments: YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch, YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238075#comment-14238075 ] Mit Desai commented on YARN-2900: - Thanks. I will check that out. > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Mit Desai > Attachments: YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch, YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2900: Attachment: YARN-2900.patch Thanks for taking a look and providing useful comments [~jeagles] and [~zjshen]. I have attached the new patch. Please take a look and let me know if this looks good. > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Mit Desai > Attachments: YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch, YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2900: Attachment: YARN-2900.patch Refining the patch > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Mit Desai > Attachments: YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch, YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2900: Attachment: YARN-2900.patch [~zjshen], [~jeagles]: Attaching final patch with the fix and unit tests to verify it. Can you review? > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Mit Desai > Attachments: YARN-2900.patch, YARN-2900.patch, YARN-2900.patch, > YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14230691#comment-14230691 ] Mit Desai commented on YARN-2890: - Verified that the failures are not caused by the patch. > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 2.7.0 > > Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, > YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2900) Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)
[ https://issues.apache.org/jira/browse/YARN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2900: Attachment: YARN-2900.patch [~zjshen] I am still working on the unit test. Meanwhile would you like to take a look at the changes that you suggested? > Application (Attempt and Container) Not Found in AHS results in Internal > Server Error (500) > --- > > Key: YARN-2900 > URL: https://issues.apache.org/jira/browse/YARN-2900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Mit Desai > Attachments: YARN-2900.patch, YARN-2900.patch, YARN-2900.patch > > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.convertToApplicationReport(ApplicationHistoryManagerImpl.java:128) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getApplication(ApplicationHistoryManagerImpl.java:118) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:222) > at > org.apache.hadoop.yarn.server.webapp.WebServices$2.run(WebServices.java:219) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679) > at > org.apache.hadoop.yarn.server.webapp.WebServices.getApp(WebServices.java:218) > ... 59 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2890: Attachment: YARN-2890.patch Attaching the patch that includes a MR job running with timeline service disabled. Also cleaned up couple debug statements. > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 2.7.0 > > Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch, > YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14230422#comment-14230422 ] Mit Desai commented on YARN-2890: - Tried running these tests without the patch. They still fail. Does not look like a side effect of this patch. bq. Would you please run through an MR job to validate the job can run successfully? Nice catch. I thought this scenario was already covered. Will update the patch. > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 2.7.0 > > Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[
https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14230420#comment-14230420
]
Mit Desai commented on YARN-2890:
-
{quote}
-1 core tests. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests:
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
{quote}
> MiniMRYarnCluster should turn on timeline service if configured to do so
>
>
> Key: YARN-2890
> URL: https://issues.apache.org/jira/browse/YARN-2890
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Mit Desai
>Assignee: Mit Desai
> Fix For: 2.7.0
>
> Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch
>
>
> Currently the MiniMRYarnCluster does not consider the configuration value for
> enabling timeline service before starting. The MiniYarnCluster should only
> start the timeline service if it is configured to do so.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2890) MiniMRYarnCluster should turn on timeline service if configured to do so
[ https://issues.apache.org/jira/browse/YARN-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated YARN-2890: Attachment: YARN-2890.patch Attaching the updated patch > MiniMRYarnCluster should turn on timeline service if configured to do so > > > Key: YARN-2890 > URL: https://issues.apache.org/jira/browse/YARN-2890 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 2.6.1 > > Attachments: YARN-2890.patch, YARN-2890.patch, YARN-2890.patch > > > Currently the MiniMRYarnCluster does not consider the configuration value for > enabling timeline service before starting. The MiniYarnCluster should only > start the timeline service if it is configured to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
