Hi Ketan, Is your Cube data from Hive or Kafka?
The line number seems couldn't match with the Kylin 2.2 source code; Can you double confirm the Kylin version? In Kylin installation folder, there is a commit id sha file, please provide that file content to us so we can exactly match the line numbers. Thanks. 2017-12-22 18:12 GMT+08:00 ShaoFeng Shi <[email protected]>: > It should be a bug; I took a quick look but didn't find the root cause > (very busy in the year-end). Could you please investigate it and then > contribute a patch? Thanks! > > 2017-12-22 16:41 GMT+08:00 ketan dikshit <[email protected]>: > >> Hi Guys, >> Any update on this, as I see this is a bug or ? >> >> We are passing in null as TSRange and the check >> Preconditions.checkArgument(tsRange != null); is always failing. >> >> Help would be appreciated. >> >> Thanks, >> Ketan >> >> > On 21-Dec-2017, at 3:06 PM, ketan dikshit <[email protected]> >> wrote: >> > >> > Hi Guys, >> > I am using Kylin 2.2 and getting some issues with Kylin auto merge. >> > The error log in kylin.log is here: >> > >> > 2017-12-21 08:31:18,811 INFO [Thread-50937] service.CubeService:552 : >> checking keepCubeRetention >> > 2017-12-21 08:31:18,811 DEBUG [Thread-50937] model.Segments:201 : Cube >> publishers_v4 has 1 building segments >> > 2017-12-21 08:31:18,811 ERROR [Thread-50937] service.CacheService:87 : >> Error in updateOnNewSegmentReady() >> > java.lang.IllegalArgumentException >> > at com.google.common.base.Preconditions.checkArgument(Precondit >> ions.java:76) >> > at org.apache.kylin.cube.CubeManager.mergeSegments(CubeManager. >> java:544) >> > at org.apache.kylin.rest.service.CubeService.mergeCubeSegment(C >> ubeService.java:601) >> > at org.apache.kylin.rest.service.CubeService.updateOnNewSegment >> Ready(CubeService.java:545) >> > at org.apache.kylin.rest.service.CubeService$$FastClassBySpring >> CGLIB$$17a07c0e.invoke(<generated>) >> > at org.springframework.cglib.proxy.MethodProxy.invoke(MethodPro >> xy.java:204) >> > at org.springframework.aop.framework.CglibAopProxy$DynamicAdvis >> edInterceptor.intercept(CglibAopProxy.java:669) >> > at org.apache.kylin.rest.service.CubeService$$EnhancerBySpringC >> GLIB$$e4f6b188.updateOnNewSegmentReady(<generated>) >> > at org.apache.kylin.rest.service.CacheService$1$1.run(CacheServ >> ice.java:85) >> > >> > >> > Auto Merge Thresholds >> > 8 (Hours) >> > 1 (Days) >> > 7 (Days) >> > 15 (Days) >> > >> > >> > As I see inside the code, I see, >> > private void mergeCubeSegment(String cubeName) { >> > CubeInstance cube = getCubeManager().getCube(cubeName); >> > if (!cube.needAutoMerge()) { >> > return; >> > } >> > >> > synchronized (CubeService.class) { >> > try { >> > cube = getCubeManager().getCube(cubeName); >> > SegmentRange offsets = cube.autoMergeCubeSegments(); >> > if (offsets != null) { >> > CubeSegment newSeg = getCubeManager().mergeSegments(cube, >> null, offsets, true); >> > logger.debug("Will submit merge job on " + newSeg); >> > DefaultChainedExecutable job = >> EngineFactory.createBatchMergeJob(newSeg, "SYSTEM"); >> > getExecutableManager().addJob(job); >> > } else { >> > logger.debug("Not ready for merge on cube " + cubeName); >> > } >> > } catch (IOException e) { >> > logger.error("Failed to auto merge cube " + cubeName, e); >> > } >> > } >> > } >> > >> > ie; getCubeManager().mergeSegments(cube, null, offsets, true); is >> passing the TSRange as ‘null’; >> > to CubeManager:public CubeSegment mergeSegments(CubeInstance cube, >> TSRange tsRange, SegmentRange segRange, boolean force) method; >> > which is defined as: >> > if (cube.getSegments().isEmpty()) >> > throw new IllegalArgumentException("Cube " + cube + " has no >> segments"); >> > >> > checkInputRanges(tsRange, segRange); >> > checkBuildingSegment(cube); >> > checkCubeIsPartitioned(cube); >> > >> > if (cube.getSegments().getFirstSegment().isOffsetCube()) { >> > // offset cube, merge by date range? >> > if (segRange == null && tsRange != null) { >> > Pair<CubeSegment, CubeSegment> pair = >> cube.getSegments(SegmentStatusEnum.READY) >> > .findMergeOffsetsByDateRange(tsRange, Long.MAX_VALUE); >> > if (pair == null) >> > throw new IllegalArgumentException("Find no segments to >> merge by " + tsRange + " for cube " + cube); >> > segRange = new SegmentRange(pair.getFirst().getSegRange().start, >> pair.getSecond().getSegRange().end); >> > } >> > tsRange = null; >> > Preconditions.checkArgument(segRange != null); >> > } else { >> > segRange = null; >> > Preconditions.checkArgument(tsRange != null); >> > } >> > >> > In my case, this is getting invoked, >> > else { >> > segRange = null; >> > Preconditions.checkArgument(tsRange != null); >> > } >> > >> > and as we have passed in tsRange as NULL , so we get this exception: >> > java.lang.IllegalArgumentException >> > at com.google.common.base.Preconditions.checkArgument(Precondit >> ions.java:76) >> > at org.apache.kylin.cube.CubeManager.mergeSegments(CubeManager. >> java:544) >> > at org.apache.kylin.rest.service.CubeService.mergeCubeSegment(C >> ubeService.java:601) >> > at org.apache.kylin.rest.service.CubeService.updateOnNewSegment >> Ready(CubeService.java:545) >> > at org.apache.kylin.rest.service.CubeService$$FastClassBySpring >> CGLIB$$17a07c0e.invoke(<generated>) >> > at org.springframework.cglib.proxy.MethodProxy.invoke(MethodPro >> xy.java:204) >> > at org.springframework.aop.framework.CglibAopProxy$DynamicAdvis >> edInterceptor.intercept(CglibAopProxy.java:669) >> > at org.apache.kylin.rest.service.CubeService$$EnhancerBySpringC >> GLIB$$e4f6b188.updateOnNewSegmentReady(<generated>) >> > at org.apache.kylin.rest.service.CacheService$1$1.run(CacheServ >> ice.java:85) >> > >> > And the auto merge fails. >> > >> > Let me know in case I am missing on something here, or a fix maybe. >> > >> > >> > Thanks, >> > Ketan >> > >> > >> > >> > >> >> > > > -- > Best regards, > > Shaofeng Shi 史少锋 > > -- Best regards, Shaofeng Shi 史少锋
