Re: Kylin comprehensive debug documentation

[you Zhuang]

Hi Shaofeng, I think Inmem cubing code are so complicated, I wade through but 
get little. 

1. In InMemCubeBuilder.java , line 276: 

CuboidResult newCuboid = buildCuboid(task.parent, task.childCuboidId);
addChildTasks(newCuboid);

 Does this buildCuboid method output build results to ConcurrentDiskStore file 
? If so, concurrent write to the same ConcurrentDiskStore file causes 

 dirty records in Simultaneous jobs . And the blog 
http://kylin.apache.org/blog/2015/08/15/fast-cubing/  describing “ As “*” has 
no child, it will be 

outputted once be calculated, then comes “D”; After “C” be outputted, “CD” will 
be outputted as all of its children has 

finished; “is not correct.  it output parent cuboid then add child cuboids 
in above code .  output order should be ABCD->BCD->CD->D->* 。


2. In  BatchCubingJobBuilder2.java, line 122:

final int maxLevel = 
CuboidUtil.getLongestDepth(seg.getCuboidScheduler().getAllCuboidIds());

Is this some more complicated?  Why not use 
seg.getCuboidScheduler().getBuildLevel());   I see this snippet at line 71 in 
CuboidJob.java
 

 

> On Oct 31, 2018, at 3:52 PM, ShaoFeng Shi  wrote:
> 
> Thank you Jiatao!
> 
> Dear users, if you see the document has a problem, please point it out with 
> email or JIRA, or fire a github PR to the "document" branch. We will continue 
> to refine the docs!
> 
> JiaTao Tao  于2018年10月31日周三 下午3:48写道：
> You are welcome :).
> 
> you Zhuang  于2018年10月31日周三 上午10:11写道：
> Much appreciation to JiaTao, bower install failed before , when I open the 
> proxy , all downloads  succeed.  Now everything goes well.
> 
>> On Oct 31, 2018, at 9:46 AM, JiaTao Tao  wrote:
>> 
>> bower --allow-root install
>> 
> 
> 
> 
> -- 
> 
> Regards!
> Aron Tao
> 
> 
> -- 
> Best regards,
> 
> Shaofeng Shi 史少锋
> 

Re: Kylin comprehensive debug documentation

[JiaTao Tao]

You are welcome :).

you Zhuang  于2018年10月31日周三 上午10:11写道：

> Much appreciation to JiaTao, bower install failed before , when I open the
> proxy , all downloads  succeed.  Now everything goes well.
>
> On Oct 31, 2018, at 9:46 AM, JiaTao Tao  wrote:
>
> bower --allow-root install
>
>
>
>

-- 


Regards!

Aron Tao

Re: Kylin comprehensive debug documentation

[you Zhuang]

Much appreciation to JiaTao, bower install failed before , when I open the 
proxy , all downloads  succeed.  Now everything goes well.

> On Oct 31, 2018, at 9:46 AM, JiaTao Tao  wrote:
> 
> bower --allow-root install
> 

Re: Kylin comprehensive debug documentation

[you Zhuang]

Yeah, I have done above steps, but I think that’s not enough, the final step 
‘bower install ‘ just installs some js modules.

> On Oct 31, 2018, at 9:46 AM, JiaTao Tao  wrote:
> 
> bower --allow-root install
> 

Re: Kylin comprehensive debug documentation

[JiaTao Tao]

Did you do these mentioned in the doc?

Launch Kylin Web Server locally

Copy server/src/main/webapp/WEB-INF to webapp/app/WEB-INF


cp -r server/src/main/webapp/WEB-INF webapp/app/WEB-INF

Download JS for Kylin web GUI. npm is part of Node.js, please search about
how to install it on your OS.

cd webapp

npm install -g bower

bower --allow-root install

Note, if on Windows, after install bower, need to add the path of
“bower.cmd” to system environment variable ‘PATH’, and then run:

bower.cmd --allow-root instal


you Zhuang  于2018年10月30日周二 下午9:37写道：

> Hi JiaTao, I operate according to
> http://kylin.apache.org/development/dev_env.html, but css and js becomes
> 404. Such as
> http://localhost:7070/kylin/components/bootstrap/dist/css/bootstrap.min.css,
>  What is webapp directory like?
>
> Do I need execute “grunt dev --buildEnv=dev --buildNumber=`date
> "+%Y%m%d%H%M%S"` || { exit 1; }"
>
> When I execute above command , a dist directory generated. But what should
> I do .
>
> I have little knowledge about nodejs ,  is there any other commands must
> be executed?
>
> On Oct 30, 2018, at 2:56 PM, JiaTao Tao  wrote:
>
> You cannot debug MR job directly in DebugTomcat. In fact, it might be
> better adding more logs for tracing, in my opinion.
>
> If you do need to debug MR job directly, you may need to change your
> core-site.xml, mapred-site.xml and etc. to run MR job locally, It is
> theoretically possible, you can give a try :).
>
> you Zhuang  于2018年10月30日周二 下午2:23写道：
>
>> And also debug map reduce job ?
>>
>> On Oct 30, 2018, at 12:51 PM, JiaTao Tao  wrote:
>>
>> You may need this: http://kylin.apache.org/development/dev_env.html.
>> Using debug mode to launch org.apache.kylin.rest.DebugTomcat, then you
>> can debug Kylin server
>>
>> --
>>
>> Regards!
>> Aron Tao
>>
>> you Zhuang  于2018年10月30日周二 下午12:04写道：
>>
>>> 1. I can debug Kylin and standalone mr job  in intellij idea with
>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9001 , but I
>>> don’t know how to debug both Kylin web server and whole build job flow,
>>> which mean idea can’t set up two remote debug servers
>>>
>>> 2. can we write a comprehensive documentation about full debug
>>> situations ?  Because without debugging everything becomes awkward .
>>
>>
>>
>>
>>
>>
>
> --
>
> Regards!
> Aron Tao
>
>
>

-- 


Regards!

Aron Tao

Re: Kylin comprehensive debug documentation

[you Zhuang]

Hi JiaTao, I operate according to 
http://kylin.apache.org/development/dev_env.html 
, but css and js becomes 404. 
Such as 
http://localhost:7070/kylin/components/bootstrap/dist/css/bootstrap.min.css 
,  
What is webapp directory like?

Do I need execute “grunt dev --buildEnv=dev --buildNumber=`date 
"+%Y%m%d%H%M%S"` || { exit 1; }"

When I execute above command , a dist directory generated. But what should I do 
.

I have little knowledge about nodejs ,  is there any other commands must be 
executed?

> On Oct 30, 2018, at 2:56 PM, JiaTao Tao  wrote:
> 
> You cannot debug MR job directly in DebugTomcat. In fact, it might be better 
> adding more logs for tracing, in my opinion. 
> 
> If you do need to debug MR job directly, you may need to change your 
> core-site.xml, mapred-site.xml and etc. to run MR job locally, It is 
> theoretically possible, you can give a try :).
> 
> you Zhuang mailto:zhuangzixiao...@gmail.com>> 
> 于2018年10月30日周二 下午2:23写道：
> And also debug map reduce job ?
> 
>> On Oct 30, 2018, at 12:51 PM, JiaTao Tao > > wrote:
>> 
>> You may need this: http://kylin.apache.org/development/dev_env.html 
>> .
>> Using debug mode to launch org.apache.kylin.rest.DebugTomcat, then you can 
>> debug Kylin server
>> 
>> -- 
>> 
>> Regards!
>> Aron Tao
>> 
>> you Zhuang mailto:zhuangzixiao...@gmail.com>> 
>> 于2018年10月30日周二 下午12:04写道：
>> 1. I can debug Kylin and standalone mr job  in intellij idea with 
>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9001 , but I 
>> don’t know how to debug both Kylin web server and whole build job flow, 
>> which mean idea can’t set up two remote debug servers 
>> 
>> 2. can we write a comprehensive documentation about full debug situations ?  
>> Because without debugging everything becomes awkward .
>> 
>> 
>> 
> 
> 
> 
> -- 
> 
> Regards!
> Aron Tao

Re: Kylin comprehensive debug documentation

[you Zhuang]

I think your description is no difference with I debug with remote server and 
one by one debugging mr job 

> On Oct 30, 2018, at 2:56 PM, JiaTao Tao  wrote:
> 
> You cannot debug MR job directly in DebugTomcat. In fact, it might be better 
> adding more logs for tracing, in my opinion. 
> 
> If you do need to debug MR job directly, you may need to change your 
> core-site.xml, mapred-site.xml and etc. to run MR job locally, It is 
> theoretically possible, you can give a try :).
> 
> you Zhuang mailto:zhuangzixiao...@gmail.com>> 
> 于2018年10月30日周二 下午2:23写道：
> And also debug map reduce job ?
> 
>> On Oct 30, 2018, at 12:51 PM, JiaTao Tao > > wrote:
>> 
>> You may need this: http://kylin.apache.org/development/dev_env.html 
>> .
>> Using debug mode to launch org.apache.kylin.rest.DebugTomcat, then you can 
>> debug Kylin server
>> 
>> -- 
>> 
>> Regards!
>> Aron Tao
>> 
>> you Zhuang mailto:zhuangzixiao...@gmail.com>> 
>> 于2018年10月30日周二 下午12:04写道：
>> 1. I can debug Kylin and standalone mr job  in intellij idea with 
>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9001 , but I 
>> don’t know how to debug both Kylin web server and whole build job flow, 
>> which mean idea can’t set up two remote debug servers 
>> 
>> 2. can we write a comprehensive documentation about full debug situations ?  
>> Because without debugging everything becomes awkward .
>> 
>> 
>> 
> 
> 
> 
> -- 
> 
> Regards!
> Aron Tao

Re: Kylin comprehensive debug documentation

[George Ni]

Besides, you can write test cases for kylin's MR jobs with mrunit to debug, see 
org.apache.kylin.engine.mr.steps.NDCuboidMapperTest for reference.

Best regards,
 
Chun’en Ni(George)

- 原始邮件 -
发件人: JiaTao Tao 
收件人: user@kylin.apache.org
已发送邮件: Tue, 30 Oct 2018 14:56:54 +0800 (CST)
主题: Re: Kylin comprehensive debug documentation

You cannot debug MR job directly in DebugTomcat. In fact, it might be better 
adding more logs for tracing, in my opinion. 
If you do need to debug MR job directly, you may need to change your 
core-site.xml, mapred-site.xml and etc. to run MR job locally, It is 
theoretically possible, you can give a try :).
you Zhuang  于2018年10月30日周二 下午2:23写道：
And also debug map reduce job ?

On Oct 30, 2018, at 12:51 PM, JiaTao Tao  wrote:You may 
need this: http://kylin.apache.org/development/dev_env.html.Using debug mode to 
launch org.apache.kylin.rest.DebugTomcat, then you can debug Kylin server

-- 

Regards!Aron Tao
you Zhuang  于2018年10月30日周二 下午12:04写道：
1. I can debug Kylin and standalone mr job  in intellij idea with 
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9001 , but I 
don’t know how to debug both Kylin web server and whole build job flow, which 
mean idea can’t set up two remote debug servers 


2. can we write a comprehensive documentation about full debug situations ?  
Because without debugging everything becomes awkward .



-- 

Regards!Aron Tao

Re: Kylin comprehensive debug documentation

[JiaTao Tao]

You cannot debug MR job directly in DebugTomcat. In fact, it might be
better adding more logs for tracing, in my opinion.

If you do need to debug MR job directly, you may need to change your
core-site.xml, mapred-site.xml and etc. to run MR job locally, It is
theoretically possible, you can give a try :).

you Zhuang  于2018年10月30日周二 下午2:23写道：

> And also debug map reduce job ?
>
> On Oct 30, 2018, at 12:51 PM, JiaTao Tao  wrote:
>
> You may need this: http://kylin.apache.org/development/dev_env.html.
> Using debug mode to launch org.apache.kylin.rest.DebugTomcat, then you can
> debug Kylin server
>
> --
>
> Regards!
> Aron Tao
>
> you Zhuang  于2018年10月30日周二 下午12:04写道：
>
>> 1. I can debug Kylin and standalone mr job  in intellij idea with
>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9001 , but I
>> don’t know how to debug both Kylin web server and whole build job flow,
>> which mean idea can’t set up two remote debug servers
>>
>> 2. can we write a comprehensive documentation about full debug situations
>> ?  Because without debugging everything becomes awkward .
>
>
>
>
>
>

-- 


Regards!

Aron Tao

Re: Kylin comprehensive debug documentation

[you Zhuang]

And also debug map reduce job ?

> On Oct 30, 2018, at 12:51 PM, JiaTao Tao  wrote:
> 
> You may need this: http://kylin.apache.org/development/dev_env.html 
> .
> Using debug mode to launch org.apache.kylin.rest.DebugTomcat, then you can 
> debug Kylin server
> 
> -- 
> 
> Regards!
> Aron Tao
> 
> you Zhuang mailto:zhuangzixiao...@gmail.com>> 
> 于2018年10月30日周二 下午12:04写道：
> 1. I can debug Kylin and standalone mr job  in intellij idea with 
> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9001 , but I 
> don’t know how to debug both Kylin web server and whole build job flow, which 
> mean idea can’t set up two remote debug servers 
> 
> 2. can we write a comprehensive documentation about full debug situations ?  
> Because without debugging everything becomes awkward .
> 
> 
> 

Re: Kylin comprehensive debug documentation

[JiaTao Tao]

You may need this: http://kylin.apache.org/development/dev_env.html.
Using debug mode to launch org.apache.kylin.rest.DebugTomcat, then you can
debug Kylin server

-- 


Regards!

Aron Tao

you Zhuang  于2018年10月30日周二 下午12:04写道：

> 1. I can debug Kylin and standalone mr job  in intellij idea with
> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9001 , but I
> don’t know how to debug both Kylin web server and whole build job flow,
> which mean idea can’t set up two remote debug servers
>
> 2. can we write a comprehensive documentation about full debug situations
> ?  Because without debugging everything becomes awkward .