The easiest is to discard the broken job and re-build the cube. Disable the cube, purge it, then build again -- these 3 steps often fix corrupt cube.
On Thu, Jul 16, 2015 at 9:53 AM, Shi, Shaofeng <[email protected]> wrote: > From the source code of hbase HFileOutputFormat2.java, it tries to create > file in ³/tmp² (HDFS path); you may need have a need on your HDFS: > > static void configurePartitioner(Job job, List<ImmutableBytesWritable> > splitPoints) > throws IOException { > > // create the partitions file > FileSystem fs = FileSystem.get(job.getConfiguration()); > Path partitionsPath = new Path("/tmp", "partitions_" + > UUID.randomUUID()); > > > > > > On 7/15/15, 11:07 PM, "alex schufo" <[email protected]> wrote: > > >Yes, my installation was apparently corrupted. > >I created a tmp folder and the build first steps work but now step #10 > >fails with > > > >java.lang.IllegalArgumentException: Can not create a Path from a null > >string > > at org.apache.hadoop.fs.Path.checkPathArg(Path.java:123) > > at org.apache.hadoop.fs.Path.<init>(Path.java:135) > > at org.apache.hadoop.fs.Path.<init>(Path.java:89) > > at > >org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configurePartitioner( > >HFileOutputFormat2.java:545) > > at > >org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalL > >oad(HFileOutputFormat2.java:394) > > at > >org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureIncrementalLo > >ad(HFileOutputFormat.java:88) > > at > >org.apache.kylin.job.hadoop.cube.CubeHFileJob.run(CubeHFileJob.java:89) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > > at > >org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable > >.java:112) > > at > >org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutab > >le.java:107) > > at > >org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChai > >nedExecutable.java:50) > > at > >org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutab > >le.java:107) > > at > >org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(Defaul > >tScheduler.java:132) > > at > >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > >1145) > > at > >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java > >:615) > > at java.lang.Thread.run(Thread.java:744) > > > > > >On Wed, Jul 15, 2015 at 2:44 PM, Shi, Shaofeng <[email protected]> wrote: > > > >> No, no need; All metadata and cubes are in Hbase/hdfs with redundancy; > >> Local disk is just using for temp files; > >> > >> On 7/15/15, 8:35 PM, "alex schufo" <[email protected]> wrote: > >> > >> >Do I have to re-create completely new cubes? > >> > > >> >On Wed, Jul 15, 2015 at 1:34 PM, alex schufo <[email protected]> > >> wrote: > >> > > >> >> I am in a situation where the machine where Kylin was installed went > >> >>down. > >> >> The Hadoop and HBase clusters are still working normally with the > >>Kylin > >> >> metadata still there. > >> >> > >> >> I installed Kylin on a new machine and I was assuming that thanks to > >>the > >> >> metadata it could work as before. > >> >> > >> >> It worked to an extent: I can access Kylin, see my existing projects, > >> >>see > >> >> my previous cubes and jobs. > >> >> > >> >> However if I go to the query panel of a cube I cannot see the Table > >>and > >> >>in > >> >> the log I can see : > >> >> > >> >> > >> >> [http-bio-7070-exec-6]:[2015-07-15 > >> > >>>>05:33:06,917][ERROR][org.apache.kylin.rest.controller.BasicController.h > >>>>an > >> >>dleError(BasicController.java:52)] > >> >> - > >> >> > >> >> java.lang.RuntimeException: java.io.IOException: No such file or > >> >>directory > >> >> > >> >> at > >> >> > >> > >>>>org.apache.kylin.query.schema.OLAPSchemaFactory.createTempOLAPJson(OLAP > >>>>Sc > >> >>hemaFactory.java:134) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.kylin.rest.service.BasicService.getOLAPDataSource(BasicServi > >>>>ce > >> >>.java:103) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.kylin.rest.service.QueryService.getMetadata(QueryService.jav > >>>>a: > >> >>285) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.kylin.rest.service.QueryService.getMetadata(QueryService.jav > >>>>a: > >> >>104) > >> >> > >> >> at > >> >> > >> >>org.apache.kylin.rest.service.QueryService$$FastClassByCGLIB$$ > >> 4957273f.in > >> >>voke(<generated>) > >> >> > >> >> at > >>net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedIntercep > >>>>to > >> >>r.intercept(Cglib2AopProxy.java:618) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.kylin.rest.service.QueryService$$EnhancerByCGLIB$$ > 844b396.ge > >>>>tM > >> >>etadata(<generated>) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.kylin.rest.controller.QueryController.getMetadata(QueryContr > >>>>ol > >> >>ler.java:179) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.kylin.rest.controller.QueryController$$FastClassByCGLIB$$fc0 > >>>>39 > >> >>d0b.invoke(<generated>) > >> >> > >> >> at > >>net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedIntercep > >>>>to > >> >>r.intercept(Cglib2AopProxy.java:618) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.kylin.rest.controller.QueryController$$EnhancerByCGLIB$$595f > >>>>f2 > >> >>c9.getMetadata(<generated>) > >> >> > >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >>Method) > >> >> > >> >> at > >> >> > >> > >>>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja > >>>>va > >> >>:57) > >> >> > >> >> at > >> >> > >> > >>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso > >>>>rI > >> >>mpl.java:43) > >> >> > >> >> at java.lang.reflect.Method.invoke(Method.java:606) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.web.method.support.InvocableHandlerMethod.invoke(In > >>>>vo > >> >>cableHandlerMethod.java:213) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.web.method.support.InvocableHandlerMethod.invokeFor > >>>>Re > >> >>quest(InvocableHandlerMethod.java:126) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.web.servlet.mvc.method.annotation.ServletInvocableH > >>>>an > >> >>dlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.web.servlet.mvc.method.annotation.RequestMappingHan > >>>>dl > >> >>erAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.web.servlet.mvc.method.annotation.RequestMappingHan > >>>>dl > >> >>erAdapter.handleInternal(RequestMappingHandlerAdapter.java:578) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter > >>>>.h > >> >>andle(AbstractHandlerMethodAdapter.java:80) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.web.servlet.DispatcherServlet.doDispatch(Dispatcher > >>>>Se > >> >>rvlet.java:923) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.web.servlet.DispatcherServlet.doService(DispatcherS > >>>>er > >> >>vlet.java:852) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.web.servlet.FrameworkServlet.processRequest(Framewo > >>>>rk > >> >>Servlet.java:882) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet > >>>>.j > >> >>ava:778) > >> >> > >> >> at > >>javax.servlet.http.HttpServlet.service(HttpServlet.java:620) > >> >> > >> >> at > >>javax.servlet.http.HttpServlet.service(HttpServlet.java:727) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applic > >>>>at > >> >>ionFilterChain.java:303) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFil > >>>>te > >> >>rChain.java:208) > >> >> > >> >> at > >> >> > >>org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applic > >>>>at > >> >>ionFilterChain.java:241) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFil > >>>>te > >> >>rChain.java:208) > >> >> > >> >> at > >> >> > >> > >>>>com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(Abstra > >>>>ct > >> >>InstrumentedFilter.java:97) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applic > >>>>at > >> >>ionFilterChain.java:241) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFil > >>>>te > >> >>rChain.java:208) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.do > >>>>Fi > >> >>lter(FilterChainProxy.java:330) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.access.intercept.FilterSecurityInterce > >>>>pt > >> >>or.invoke(FilterSecurityInterceptor.java:118) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.access.intercept.FilterSecurityInterce > >>>>pt > >> >>or.doFilter(FilterSecurityInterceptor.java:84) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.do > >>>>Fi > >> >>lter(FilterChainProxy.java:342) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.access.ExceptionTranslationFilter.doFi > >>>>lt > >> >>er(ExceptionTranslationFilter.java:113) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.do > >>>>Fi > >> >>lter(FilterChainProxy.java:342) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.session.SessionManagementFilter.doFilt > >>>>er > >> >>(SessionManagementFilter.java:103) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.do > >>>>Fi > >> >>lter(FilterChainProxy.java:342) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.authentication.AnonymousAuthentication > >>>>Fi > >> >>lter.doFilter(AnonymousAuthenticationFilter.java:113) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.do > >>>>Fi > >> >>lter(FilterChainProxy.java:342) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.servletapi.SecurityContextHolderAwareR > >>>>eq > >> >>uestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.do > >>>>Fi > >> >>lter(FilterChainProxy.java:342) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.savedrequest.RequestCacheAwareFilter.d > >>>>oF > >> >>ilter(RequestCacheAwareFilter.java:45) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.do > >>>>Fi > >> >>lter(FilterChainProxy.java:342) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.authentication.www.BasicAuthentication > >>>>Fi > >> >>lter.doFilter(BasicAuthenticationFilter.java:150) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.do > >>>>Fi > >> >>lter(FilterChainProxy.java:342) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.authentication.ui.DefaultLoginPageGene > >>>>ra > >> >>tingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:91) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.do > >>>>Fi > >> >>lter(FilterChainProxy.java:342) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.authentication.AbstractAuthenticationP > >>>>ro > >> > >>>>cessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.do > >>>>Fi > >> >>lter(FilterChainProxy.java:342) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.authentication.logout.LogoutFilter.doF > >>>>il > >> >>ter(LogoutFilter.java:105) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.do > >>>>Fi > >> >>lter(FilterChainProxy.java:342) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.context.SecurityContextPersistenceFilt > >>>>er > >> >>.doFilter(SecurityContextPersistenceFilter.java:87) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.FilterChainProxy$VirtualFilterChain.do > >>>>Fi > >> >>lter(FilterChainProxy.java:342) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.FilterChainProxy.doFilterInternal(Filt > >>>>er > >> >>ChainProxy.java:192) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.security.web.FilterChainProxy.doFilter(FilterChainP > >>>>ro > >> >>xy.java:160) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(Del > >>>>eg > >> >>atingFilterProxy.java:346) > >> >> > >> >> at > >> >> > >> > >>>>org.springframework.web.filter.DelegatingFilterProxy.doFilter(Delegatin > >>>>gF > >> >>ilterProxy.java:259) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applic > >>>>at > >> >>ionFilterChain.java:241) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFil > >>>>te > >> >>rChain.java:208) > >> >> > >> >> at > >> >> > >>com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:205) > >> >> > >> >> at > >> >> > >>com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:266) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applic > >>>>at > >> >>ionFilterChain.java:241) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFil > >>>>te > >> >>rChain.java:208) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVal > >>>>ve > >> >>.java:220) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.core.StandardContextValve.invoke(StandardContextVal > >>>>ve > >> >>.java:122) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticato > >>>>rB > >> >>ase.java:504) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.jav > >>>>a: > >> >>170) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.jav > >>>>a: > >> >>103) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:95 > >>>>0) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve > >>>>.j > >> >>ava:116) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java: > >>>>42 > >> >>1) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11 > >>>>Pr > >> >>ocessor.java:1074) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Ab > >>>>st > >> >>ractProtocol.java:611) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint. > >>>>ja > >> >>va:314) > >> >> > >> >> at > >> >> > >> > >>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja > >>>>va > >> >>:1145) > >> >> > >> >> at > >> >> > >> > >>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j > >>>>av > >> >>a:615) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThre > >>>>ad > >> >>.java:61) > >> >> > >> >> at java.lang.Thread.run(Thread.java:744) > >> >> > >> >> Caused by: java.io.IOException: No such file or directory > >> >> > >> >> at java.io.UnixFileSystem.createFileExclusively(Native > >>Method) > >> >> > >> >> at java.io.File.createNewFile(File.java:1006) > >> >> > >> >> at java.io.File.createTempFile(File.java:1989) > >> >> > >> >> at java.io.File.createTempFile(File.java:2040) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.kylin.query.schema.OLAPSchemaFactory.createTempOLAPJson(OLAP > >>>>Sc > >> >>hemaFactory.java:96) > >> >> > >> >> ... 85 more > >> >> > >> >> If I try to build an existing cube it fails on Step #2 with this > >> >>exception: > >> >> > >> >> pool-5-thread-2]:[2015-07-15 > >> > >>>>05:08:25,256][ERROR][ > org.apache.kylin.job.common.MapReduceExecutable.do > >>>>Wo > >> >>rk(MapReduceExecutable.java:115)] > >> >> - error execute > >> >> MapReduceExecutable{id=d729fd24-bb4f-47d6-b895-1e01594c60b5-01, > >> >> name=Extract Fact Table Distinct Columns, state=RUNNING} > >> >> > >> >> java.io.IOException: No such file or directory > >> >> > >> >> at java.io.UnixFileSystem.createFileExclusively(Native > >>Method) > >> >> > >> >> at java.io.File.createNewFile(File.java:1006) > >> >> > >> >> at java.io.File.createTempFile(File.java:1989) > >> >> > >> >> at java.io.File.createTempFile(File.java:2040) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.kylin.job.hadoop.AbstractHadoopJob.attachKylinPropsAndMetada > >>>>ta > >> >>(AbstractHadoopJob.java:217) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinc > >>>>tC > >> >>olumnsJob.java:78) > >> >> > >> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > >> >> > >> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecuta > >>>>bl > >> >>e.java:112) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecu > >>>>ta > >> >>ble.java:107) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultC > >>>>ha > >> >>inedExecutable.java:50) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecu > >>>>ta > >> >>ble.java:107) > >> >> > >> >> at > >> >> > >> > >>>>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(Def > >>>>au > >> >>ltScheduler.java:132) > >> >> > >> >> at > >> >> > >> > >>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja > >>>>va > >> >>:1145) > >> >> > >> >> at > >> >> > >> > >>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j > >>>>av > >> >>a:615) > >> >> > >> >> at java.lang.Thread.run(Thread.java:744) > >> >> > >> > >> > >
