[jira] [Commented] (KYLIN-3371) Increase of time between steps after Kylin 2.3.1 update
[
https://issues.apache.org/jira/browse/KYLIN-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468202#comment-16468202
]
Shaofeng SHI commented on KYLIN-3371:
-
What's the "kylin.job.scheduler.poll-interval-second" in your 2.2
installation's kylin.properties?
The default value is 30 (seconds), so it will sleep 30 mintues before executing
next step.
> Increase of time between steps after Kylin 2.3.1 update
> ---
>
> Key: KYLIN-3371
> URL: https://issues.apache.org/jira/browse/KYLIN-3371
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.3.1
> Environment: -EMR 5.7 (S3 storage for Hive and HBase)
> -Kylin 2.3.1 (updated from 2.2). Deployed on separated EC2 instance.
>Reporter: Roberto Tardío Olmos
>Priority: Major
> Attachments: Screenshot1.png, Screenshot2.png
>
>
> Hi,
> We have updated from Kylin 2.2 to Kylin 2.3.1. We performed the update
> process with successful. However, we have experimented an increase of time
> wasted between steps during building time. For example, the time between step
> 7 “Build N-Dimension Cuboid:level 1” and “…level2” is now about 30 seconds, a
> lot of time if we take into account that in these steps no processing is
> executed since we use the "layer" algorithm (Screenshot 1). This time between
> steps is much longer than when we used Kylin 2.2, were average time was about
> 5 seconds. (Screenshot 2).
> Could it be related to a change in the code? The configuration we use in
> Kylin 2.3.1 is the same as we used in Kylin 2.2. I have considered this JIRA
> as a possible BUG.
> Thanks in advance,
>
> Screenshot 1:
> !Screenshot1.png!
>
> Screenshot 2
>
> !Screenshot2.png!
>
>
> A sample of the generated log.
> Kylin 2.2
>
> 2018-05-04 14:25:40,797 INFO [Scheduler 542798956 Job
> cb843ecc-5062-49ec-a881-215097e4501a-518] execution.ExecutableManager:421 :
> job id:cb843ecc-5062-49ec-a881-215097e4501a from RUNNING to READY
> 2018-05-04 14:25:45,496 INFO [Scheduler 542798956 Job
> cb843ecc-5062-49ec-a881-215097e4501a-518] execution.AbstractExecutable:111 :
> Executing AbstractExecutable (BUILD CUBE - Captacion_Resumido_Cubo_v3 -
> 2018050400_2018050500 - GMT+02:00 2018-05-04 14:21:34)
>
> Kylin 2.3.1
>
> 2018-05-07 15:25:26,338 INFO [Scheduler 1942482192 Job
> 6d616841-1c4d-46ca-a1b9-a18047fbada5-1006] execution.ExecutableManager:411 :
> job id:6d616841-1c4d-46ca-a1b9-a18047fbada5 from RUNNING to READY
> 2018-05-07 {color:#FF}15:25:48,365{color} INFO [Scheduler 1942482192 Job
> 6d616841-1c4d-46ca-a1b9-a18047fbada5-1006] execution.AbstractExecutable:147 :
> Executing AbstractExecutable (BUILD CUBE - Captacion_Resumido_Cubo_v3 -
> 2018050700_2018050800 - GMT+02:00 2018-05-07 15:19:41)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3366) Configure automatic enabling of cubes after a build process
[ https://issues.apache.org/jira/browse/KYLIN-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3366: Fix Version/s: v2.4.0 > Configure automatic enabling of cubes after a build process > --- > > Key: KYLIN-3366 > URL: https://issues.apache.org/jira/browse/KYLIN-3366 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.3.1 > Environment: Kylin 2.3.1 and Hadoop EMR 5.7 >Reporter: Roberto Tardío Olmos >Assignee: Pan, Julian >Priority: Minor > Labels: features > Fix For: v2.4.0 > > Attachments: KYLIN-3366.patch > > > Kylin automatically enables the disabled cubes after a construction process. > This behavior forces us to constantly disable a new cube that is under > development to replace an existing and enabled cube. If we do not disable it, > we could have problems with the routing of the queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3372) Upgrade jackson-databind version due to security concerns
Ted Yu created KYLIN-3372: - Summary: Upgrade jackson-databind version due to security concerns Key: KYLIN-3372 URL: https://issues.apache.org/jira/browse/KYLIN-3372 Project: Kylin Issue Type: Task Reporter: Ted Yu * https://nvd.nist.gov/vuln/detail/CVE-2018-5968 * https://nvd.nist.gov/vuln/detail/CVE-2018-7489 * https://nvd.nist.gov/vuln/detail/CVE-2017-7525 * https://nvd.nist.gov/vuln/detail/CVE-2017-17485 * https://nvd.nist.gov/vuln/detail/CVE-2017-15095 We should either remove the dependency or upgrade to version 2.8.11.1 or the latest, if possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3366) Configure automatic enabling of cubes after a build process
[ https://issues.apache.org/jira/browse/KYLIN-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467656#comment-16467656 ] Roberto Tardío Olmos commented on KYLIN-3366: - We have compiled your patch with Kylin 2.3.1 and works fine. In my opinion it is solved, it would be interesting to include this feature in the next version of Kylin. > Configure automatic enabling of cubes after a build process > --- > > Key: KYLIN-3366 > URL: https://issues.apache.org/jira/browse/KYLIN-3366 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.3.1 > Environment: Kylin 2.3.1 and Hadoop EMR 5.7 >Reporter: Roberto Tardío Olmos >Assignee: Pan, Julian >Priority: Minor > Labels: features > Attachments: KYLIN-3366.patch > > > Kylin automatically enables the disabled cubes after a construction process. > This behavior forces us to constantly disable a new cube that is under > development to replace an existing and enabled cube. If we do not disable it, > we could have problems with the routing of the queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3366) Configure automatic enabling of cubes after a build process
[ https://issues.apache.org/jira/browse/KYLIN-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467656#comment-16467656 ] Roberto Tardío Olmos edited comment on KYLIN-3366 at 5/8/18 4:34 PM: - We have compiled your patch with Kylin 2.3.1 and works fine. In my opinion it is solved, it would be interesting to include this feature in the next version of Kylin. Thanks Julian! was (Author: rober tardio): We have compiled your patch with Kylin 2.3.1 and works fine. In my opinion it is solved, it would be interesting to include this feature in the next version of Kylin. > Configure automatic enabling of cubes after a build process > --- > > Key: KYLIN-3366 > URL: https://issues.apache.org/jira/browse/KYLIN-3366 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.3.1 > Environment: Kylin 2.3.1 and Hadoop EMR 5.7 >Reporter: Roberto Tardío Olmos >Assignee: Pan, Julian >Priority: Minor > Labels: features > Attachments: KYLIN-3366.patch > > > Kylin automatically enables the disabled cubes after a construction process. > This behavior forces us to constantly disable a new cube that is under > development to replace an existing and enabled cube. If we do not disable it, > we could have problems with the routing of the queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3371) Increase of time between steps after Kylin 2.3.1 update
Roberto Tardío Olmos created KYLIN-3371:
---
Summary: Increase of time between steps after Kylin 2.3.1 update
Key: KYLIN-3371
URL: https://issues.apache.org/jira/browse/KYLIN-3371
Project: Kylin
Issue Type: Bug
Components: Job Engine
Affects Versions: v2.3.1
Environment: -EMR 5.7 (S3 storage for Hive and HBase)
-Kylin 2.3.1 (updated from 2.2). Deployed on separated EC2 instance.
Reporter: Roberto Tardío Olmos
Attachments: Screenshot1.png, Screenshot2.png
Hi,
We have updated from Kylin 2.2 to Kylin 2.3.1. We performed the update process
with successful. However, we have experimented an increase of time wasted
between steps during building time. For example, the time between step 7 “Build
N-Dimension Cuboid:level 1” and “…level2” is now about 30 seconds, a lot of
time if we take into account that in these steps no processing is executed
since we use the "layer" algorithm (Screenshot 1). This time between steps is
much longer than when we used Kylin 2.2, were average time was about 5 seconds.
(Screenshot 2).
Could it be related to a change in the code? The configuration we use in Kylin
2.3.1 is the same as we used in Kylin 2.2. I have considered this JIRA as a
possible BUG.
Thanks in advance,
Screenshot 1:
!Screenshot1.png!
Screenshot 2
!Screenshot2.png!
A sample of the generated log.
Kylin 2.2
2018-05-04 14:25:40,797 INFO [Scheduler 542798956 Job
cb843ecc-5062-49ec-a881-215097e4501a-518] execution.ExecutableManager:421 : job
id:cb843ecc-5062-49ec-a881-215097e4501a from RUNNING to READY
2018-05-04 14:25:45,496 INFO [Scheduler 542798956 Job
cb843ecc-5062-49ec-a881-215097e4501a-518] execution.AbstractExecutable:111 :
Executing AbstractExecutable (BUILD CUBE - Captacion_Resumido_Cubo_v3 -
2018050400_2018050500 - GMT+02:00 2018-05-04 14:21:34)
Kylin 2.3.1
2018-05-07 15:25:26,338 INFO [Scheduler 1942482192 Job
6d616841-1c4d-46ca-a1b9-a18047fbada5-1006] execution.ExecutableManager:411 :
job id:6d616841-1c4d-46ca-a1b9-a18047fbada5 from RUNNING to READY
2018-05-07 {color:#FF}15:25:48,365{color} INFO [Scheduler 1942482192 Job
6d616841-1c4d-46ca-a1b9-a18047fbada5-1006] execution.AbstractExecutable:147 :
Executing AbstractExecutable (BUILD CUBE - Captacion_Resumido_Cubo_v3 -
2018050700_2018050800 - GMT+02:00 2018-05-07 15:19:41)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3294) Remove HBaseMROutput.java, RangeKeyDistributionJob.java and other sunset classes
[ https://issues.apache.org/jira/browse/KYLIN-3294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3294. - Resolution: Fixed > Remove HBaseMROutput.java, RangeKeyDistributionJob.java and other sunset > classes > > > Key: KYLIN-3294 > URL: https://issues.apache.org/jira/browse/KYLIN-3294 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0 > > > They were legacy classes, keeping them will add maintainence effort > especially when upgrade HBase version. Should delete them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3294) Remove HBaseMROutput.java, RangeKeyDistributionJob.java and other sunset classes
[ https://issues.apache.org/jira/browse/KYLIN-3294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467195#comment-16467195 ] Shaofeng SHI commented on KYLIN-3294: - Staged in KYLIN-3294 branch, run the CI now. > Remove HBaseMROutput.java, RangeKeyDistributionJob.java and other sunset > classes > > > Key: KYLIN-3294 > URL: https://issues.apache.org/jira/browse/KYLIN-3294 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0 > > > They were legacy classes, keeping them will add maintainence effort > especially when upgrade HBase version. Should delete them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3294) Remove HBaseMROutput.java, RangeKeyDistributionJob.java and other sunset classes
[ https://issues.apache.org/jira/browse/KYLIN-3294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467063#comment-16467063 ] Shaofeng SHI commented on KYLIN-3294: - Hi Wenzheng, actually I had made the change when I compile Kylin with Hadoop 3.0 + HBase 2.0, in the master-hadoop-3.0 branch. Now I need go ahead so will take over this JIRA. > Remove HBaseMROutput.java, RangeKeyDistributionJob.java and other sunset > classes > > > Key: KYLIN-3294 > URL: https://issues.apache.org/jira/browse/KYLIN-3294 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0 > > > They were legacy classes, keeping them will add maintainence effort > especially when upgrade HBase version. Should delete them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3294) Remove HBaseMROutput.java, RangeKeyDistributionJob.java and other sunset classes
[ https://issues.apache.org/jira/browse/KYLIN-3294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-3294: --- Assignee: Shaofeng SHI (was: Wenzheng Liu) > Remove HBaseMROutput.java, RangeKeyDistributionJob.java and other sunset > classes > > > Key: KYLIN-3294 > URL: https://issues.apache.org/jira/browse/KYLIN-3294 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0 > > > They were legacy classes, keeping them will add maintainence effort > especially when upgrade HBase version. Should delete them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3351) Cube Planner not working in apche kylin 2.3.0(open Source)
[ https://issues.apache.org/jira/browse/KYLIN-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466950#comment-16466950 ] praveenece commented on KYLIN-3351: --- Ok zhong i will check jira thanks for reply > Cube Planner not working in apche kylin 2.3.0(open Source) > -- > > Key: KYLIN-3351 > URL: https://issues.apache.org/jira/browse/KYLIN-3351 > Project: Kylin > Issue Type: Task >Reporter: praveenece >Priority: Major > > Hi Team > i want test Cube planner in apache-kylin(2.3.0),So i created cube with > segment, and i hit query to cube many times more than thousands but the cube > planner there is no change in cuboid level like color changing and not get > COUNT in exactly row count and other items.can u please guide me. > Note: > Could u tell me? > Cube Planner Working only old Cube (like 3 month before) or new Cube also. > Configuration(kylin.Properties) > kylin.cube.cubeplanner.enabled=true > Kylin Version=2.3.0(SINGLE NODE) > MyCube creation =10days before > I referred: http://kylin.apache.org/docs23/howto/howto_use_cube_planner.html > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3351) Cube Planner not working in apche kylin 2.3.0(open Source)
[ https://issues.apache.org/jira/browse/KYLIN-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466937#comment-16466937 ] Zhong Yanghong commented on KYLIN-3351: --- Hi [~praveenece], there're two phases for cube planner. The first phase was triggered at the first building for a cube. In this phase, statistics is not needed. However, the second phase is cost-based which mainly depends on the collected statistics. In this phase, we can do optimization to adjust the current built cuboids and define the cuboids for future building. For more details, please refer [KYLIN-2727] > Cube Planner not working in apche kylin 2.3.0(open Source) > -- > > Key: KYLIN-3351 > URL: https://issues.apache.org/jira/browse/KYLIN-3351 > Project: Kylin > Issue Type: Task >Reporter: praveenece >Priority: Major > > Hi Team > i want test Cube planner in apache-kylin(2.3.0),So i created cube with > segment, and i hit query to cube many times more than thousands but the cube > planner there is no change in cuboid level like color changing and not get > COUNT in exactly row count and other items.can u please guide me. > Note: > Could u tell me? > Cube Planner Working only old Cube (like 3 month before) or new Cube also. > Configuration(kylin.Properties) > kylin.cube.cubeplanner.enabled=true > Kylin Version=2.3.0(SINGLE NODE) > MyCube creation =10days before > I referred: http://kylin.apache.org/docs23/howto/howto_use_cube_planner.html > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3351) Cube Planner not working in apche kylin 2.3.0(open Source)
[ https://issues.apache.org/jira/browse/KYLIN-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466935#comment-16466935 ] praveenece commented on KYLIN-3351: --- Hi Lisa , Thanks Lisa for Reply. i Didn't Run System Cube via Cron job, but i executed every cube manually. Whether its(Cron job) mandatory via execute? > Cube Planner not working in apche kylin 2.3.0(open Source) > -- > > Key: KYLIN-3351 > URL: https://issues.apache.org/jira/browse/KYLIN-3351 > Project: Kylin > Issue Type: Task >Reporter: praveenece >Priority: Major > > Hi Team > i want test Cube planner in apache-kylin(2.3.0),So i created cube with > segment, and i hit query to cube many times more than thousands but the cube > planner there is no change in cuboid level like color changing and not get > COUNT in exactly row count and other items.can u please guide me. > Note: > Could u tell me? > Cube Planner Working only old Cube (like 3 month before) or new Cube also. > Configuration(kylin.Properties) > kylin.cube.cubeplanner.enabled=true > Kylin Version=2.3.0(SINGLE NODE) > MyCube creation =10days before > I referred: http://kylin.apache.org/docs23/howto/howto_use_cube_planner.html > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
