[
https://issues.apache.org/jira/browse/KYLIN-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16334008#comment-16334008
]
Shawn Wang commented on KYLIN-3136:
-----------------------------------
HI,
Today, when I run "{{${KYLIN_HOME}/bin/kylin.sh
org.apache.kylin.tool.StorageCleanupJob --delete false}}", I found some files
are marked belonging to jobs in "ERROR" status.
But I can't find them on Kylin's monitor page, when I query the status from
API using the job_id, I got "Illegal job type" error, not sure whether this is
related to this issue, hope to be helpful and I'm looking for help how to deal
with these dangling jobs.
Thanks!
The files reported by StorageCleanupJob:
```
2018-01-22 08:08:27,172 INFO [main] job.StorageCleanupJob:144 : Skip
hdfs://cmcluster/user/kylin/kylin_metadata/kylin-24cb0188-5893-45c2-8ca3-be0c08eb90c5
from deletion list, as the path belongs to job
24cb0188-5893-45c2-8ca3-be0c08eb90c5 with status ERROR
2018-01-22 08:08:27,197 INFO [main] job.StorageCleanupJob:144 : Skip
hdfs://cmcluster/user/kylin/kylin_metadata/kylin-437eda97-55d6-4235-9a03-5cd677e4dd75
from deletion list, as the path belongs to job
437eda97-55d6-4235-9a03-5cd677e4dd75 with status ERROR
2018-01-22 08:08:27,228 INFO [main] job.StorageCleanupJob:144 : Skip
hdfs://cmcluster/user/kylin/kylin_metadata/kylin-667dc73e-5581-46c9-878b-34b7b3b64cd7
from deletion list, as the path belongs to job
667dc73e-5581-46c9-878b-34b7b3b64cd7 with status ERROR
2018-01-22 08:08:27,253 INFO [main] job.StorageCleanupJob:144 : Skip
hdfs://cmcluster/user/kylin/kylin_metadata/kylin-867df00d-8535-415b-b491-f0ae6db1bb7d
from deletion list, as the path belongs to job
867df00d-8535-415b-b491-f0ae6db1bb7d with status ERROR
2018-01-22 08:08:27,270 INFO [main] job.StorageCleanupJob:144 : Skip
hdfs://cmcluster/user/kylin/kylin_metadata/kylin-9b067fc3-3ea5-4895-98fe-a5147924e8d7
from deletion list, as the path belongs to job
9b067fc3-3ea5-4895-98fe-a5147924e8d7 with status ERROR
2018-01-22 08:08:27,309 INFO [main] job.StorageCleanupJob:144 : Skip
hdfs://cmcluster/user/kylin/kylin_metadata/kylin-b5ebebde-d962-4caa-badd-669b9e0aa944
from deletion list, as the path belongs to job
b5ebebde-d962-4caa-badd-669b9e0aa944 with status ERROR
2018-01-22 08:08:27,339 INFO [main] job.StorageCleanupJob:144 : Skip
hdfs://cmcluster/user/kylin/kylin_metadata/kylin-d874f91f-99f5-4634-82b0-f8ddc498a9fe
from deletion list, as the path belongs to job
d874f91f-99f5-4634-82b0-f8ddc498a9fe with status ERROR
```
The response from API:
```
{
"code": "999",
"data": null,
"msg": "org.apache.kylin.rest.exception.BadRequestException: Illegal job
type, id: 24cb0188-5893-45c2-8ca3-be0c08eb90c5.",
"stacktrace": "org.apache.kylin.rest.exception.InternalErrorException:
org.apache.kylin.rest.exception.BadRequestException: Illegal job type, id:
24cb0188-5893-45c2-8ca3-be0c08eb90c5.\n\tat
org.apache.kylin.rest.controller.JobController.get(JobController.java:98)\n\tat
sun.reflect.GeneratedMethodAccessor241.invoke(Unknown Source)\n\tat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
java.lang.reflect.Method.invoke(Method.java:498)\n\tat
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)\n\tat
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)\n\tat
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)\n\tat
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)\n\tat
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)\n\tat
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)\n\tat
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)\n\tat
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)\n\tat
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)\n\tat
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)\n\tat
javax.servlet.http.HttpServlet.service(HttpServlet.java:624)\n\tat
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)\n\tat
javax.servlet.http.HttpServlet.service(HttpServlet.java:731)\n\tat
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)\n\tat
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n\tat
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)\n\tat
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)\n\tat
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)\n\tat
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)\n\tat
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)\n\tat
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)\n\tat
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)\n\tat
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)\n\tat
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)\n\tat
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)\n\tat
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)\n\tat
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat
com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209)\n\tat
com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)\n\tat
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)\n\tat
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)\n\tat
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)\n\tat
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)\n\tat
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)\n\tat
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\tat
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\tat
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)\n\tat
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\tat
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)\n\tat
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\tat
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tat
java.lang.Thread.run(Thread.java:748)\nCaused by:
org.apache.kylin.rest.exception.BadRequestException: Illegal job type, id:
24cb0188-5893-45c2-8ca3-be0c08eb90c5.\n\tat
org.apache.kylin.rest.service.JobService.getSingleJobInstance(JobService.java:298)\n\tat
org.apache.kylin.rest.service.JobService.getJobInstance(JobService.java:284)\n\tat
org.apache.kylin.rest.controller.JobController.get(JobController.java:95)\n\t...
75 more\n",
"exception": "org.apache.kylin.rest.exception.BadRequestException: Illegal
job type, id: 24cb0188-5893-45c2-8ca3-be0c08eb90c5.",
"url": "http://xxxx/kylin/api/jobs/24cb0188-5893-45c2-8ca3-be0c08eb90c5";
}
```
> Endless status while subtask happens to be the illegal RUNNING
> --------------------------------------------------------------
>
> Key: KYLIN-3136
> URL: https://issues.apache.org/jira/browse/KYLIN-3136
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Reporter: Wang Cheng
> Assignee: Wang Cheng
> Priority: Minor
> Fix For: v2.3.0
>
>
> Cubing job consists of a couple of subtasks, Cubing job's status is decided
> by each subtask's situation.
> When a subtask's status is changed from RUNNING to SUCCEED, but it's output
> can not be updated successfully, which causes Cubing job reads the wrong
> child status and in endless status.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
