[
https://issues.apache.org/jira/browse/YARN-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17758798#comment-17758798
]
ASF GitHub Bot commented on YARN-11552:
---------------------------------------
hadoop-yetus commented on PR #5978:
URL: https://github.com/apache/hadoop/pull/5978#issuecomment-1692680964
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 24s | | trunk passed |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 32s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 17s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5978/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice:
The patch generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 49s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 20s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 45s | |
hadoop-yarn-server-timelineservice in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 124m 17s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5978/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5978 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b5a8e08ad428 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d44de83467df2979c9a54fa2b360e06c7be4f75f |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5978/4/testReport/ |
| Max. process+thread count | 750 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5978/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> timeline endpoint: /clusters/{clusterid}/apps/{appid}/entity-types Error when
> using hdfs store
> ----------------------------------------------------------------------------------------------
>
> Key: YARN-11552
> URL: https://issues.apache.org/jira/browse/YARN-11552
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Jiandan Yang
> Priority: Major
> Labels: pull-request-available
>
> *Problem description*
> I start hadoop cluster and timeservice based on hdfs store ，it throw
> execption when accessing
> [http://timelineserver:8088/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1692343022285],
> and detailed logs includes exception:
> {code:java}
> 2023-08-18 17:40:51 2023-08-18 09:40:51 INFO TimelineReaderWebServices:3372
> - Processed URL
> /ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo&userid=hadoop&flowrunid=1&clusterid=yjd&appid=application_1692341890246_0001
> (Took 56 ms.)
> 2023-08-18 17:40:51 2023-08-18 09:40:51 WARN GenericExceptionHandler:96 -
> INTERNAL_SERVER_ERROR
> 2023-08-18 17:40:51 javax.ws.rs.WebApplicationException:
> java.io.FileNotFoundException: File
> /tmp/hadoop-hadoop/timeline_service_data/entities/yjd/hadoop/QuasiMonteCarlo/1/application_1692341890246_0001
> does not exist.
> 2023-08-18 17:40:51 at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:200)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntityTypes(TimelineReaderWebServices.java:3368)
> 2023-08-18 17:40:51 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2023-08-18 17:40:51 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2023-08-18 17:40:51 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2023-08-18 17:40:51 at java.lang.reflect.Method.invoke(Method.java:498)
> 2023-08-18 17:40:51 at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> 2023-08-18 17:40:51 at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> 2023-08-18 17:40:51 at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> 2023-08-18 17:40:51 at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> 2023-08-18 17:40:51 at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:650)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:110)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1794)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.Server.handle(Server.java:516)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
> 2023-08-18 17:40:51 at java.lang.Thread.run(Thread.java:748)
> 2023-08-18 17:40:51 Caused by: java.io.FileNotFoundException: File
> /tmp/hadoop-hadoop/timeline_service_data/entities/yjd/hadoop/QuasiMonteCarlo/1/application_1692341890246_0001
> does not exist.
> 2023-08-18 17:40:51 at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1117)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:149)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1188)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1185)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1195)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.yarn.server.timelineservice.storage.FileSystemTimelineReaderImpl.getEntityTypes(FileSystemTimelineReaderImpl.java:442)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderManager.getEntityTypes(TimelineReaderManager.java:209)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntityTypes(TimelineReaderWebServices.java:3364)
> 2023-08-18 17:40:51 ... 70 more
> {code}
> *Problem analysis*
> The entity type write path of application is
> ${yarn.timeline-service.fs-writer.root-dir}/entities/${clusterId}/${userId}/${flowName}/${flowVersion}/${flowRun}/${appId}/${entityType}
> ,
> but read path does not include ${flowRun} part.
> This is the cause of FileNotFoundException
> *Problem solution*
> {code:java}
> changing flowRun of read path into '*', and the read path is
> ${yarn.timeline-service.fs-writer.root-dir}/entities/${clusterId}/${userId}/${flowName}/${flowVersion}/*/${appId}/${entityType}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
