[jira] [Created] (KYLIN-5763) Equivalent aggregations matches index failed when using the strategy of dimension as measure
pengfei.zhan created KYLIN-5763:
---
Summary: Equivalent aggregations matches index failed when using
the strategy of dimension as measure
Key: KYLIN-5763
URL: https://issues.apache.org/jira/browse/KYLIN-5763
Project: Kylin
Issue Type: Bug
Components: Query Engine
Affects Versions: 5.0-beta
Reporter: pengfei.zhan
Assignee: pengfei.zhan
Fix For: 5.0-beta
The following sql can not match index just has dimension lstg_format_name.
{code:sql}
select max(lstg_format_name) max1,
max( case when 1=1 then lstg_format_nameend ) max2
from test_kylin_fact{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KYLIN-5762) Initialize job scheduler encounters NPE
[ https://issues.apache.org/jira/browse/KYLIN-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831152#comment-17831152 ] pengfei.zhan commented on KYLIN-5762: - h1. Design If the parameter `kylin.job.max-concurrent-jobs` is less than 1, then creating the default thread pool with corePoolSize=1. If the parameter `kylin.job.max-concurrent-jobs` is equal or less than 0, all the tasks will be suspended. Validating the parameter value when configuring it in the page of project setting, with the value is less than 0, an error will be reported directly: illegal negative number. > Initialize job scheduler encounters NPE > --- > > Key: KYLIN-5762 > URL: https://issues.apache.org/jira/browse/KYLIN-5762 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > > Given `kylin.job.max-concurrent-jobs=0`, the initialization of job scheduler > encounters NPE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5762) Initialize job scheduler encounters NPE
[
https://issues.apache.org/jira/browse/KYLIN-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831147#comment-17831147
]
pengfei.zhan commented on KYLIN-5762:
-
h1. Root Cause
The parameter `kylin.job.max-concurrent-jobs ` can be set to 0. For example:
The value of this parameter was not 0 when KYLIN was started. Later, the user
created a batch of tasks, but the user did not want these tasks to be
scheduled, so the value of this parameter was modified to 0 in the project
parameter setting module. Next, maybe due to some problems, the user restarts
the KYLIN system and finds the system scheduled task scheduling is abnormal.
When restarting the KYLIN, the core pool size is 0, creating jobPool throws
IllegalArgumentException for the
maximumPoolSize can not be 0. Due to the initialization of Job scheduler
failed, the ExecutableContext is null. When scheduling tasks, both the jobPool
and the executableContext are null, the NPE throws.
{code:java}
// job scheduler corePoolSize
int corePoolSize = getMaxConcurrentJobLimitByProject(config, jobEngineConfig,
project);
jobPool = new ThreadPoolExecutor(corePoolSize, corePoolSize, Long.MAX_VALUE,
TimeUnit.DAYS,
new SynchronousQueue<>(), new
NamedThreadFactory("RunJobWorker(project:" + project + ")"));
context = new ExecutableContext(Maps.newConcurrentMap(),
Maps.newConcurrentMap(), jobEngineConfig.getConfig(),
epochId);{code}
On the job page we will find that all tasks are suspended. If we restart these
tasks, NPE will be thrown.
> Initialize job scheduler encounters NPE
> ---
>
> Key: KYLIN-5762
> URL: https://issues.apache.org/jira/browse/KYLIN-5762
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0-beta
>
>
> Given `kylin.job.max-concurrent-jobs=0`, the initialization of job scheduler
> encounters NPE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KYLIN-5762) Initialize job scheduler encounters NPE
pengfei.zhan created KYLIN-5762: --- Summary: Initialize job scheduler encounters NPE Key: KYLIN-5762 URL: https://issues.apache.org/jira/browse/KYLIN-5762 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta Given `kylin.job.max-concurrent-jobs=0`, the initialization of job scheduler encounters NPE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5761) Upgrade tomcat and netty to fix vulnerabilities
pengfei.zhan created KYLIN-5761: --- Summary: Upgrade tomcat and netty to fix vulnerabilities Key: KYLIN-5761 URL: https://issues.apache.org/jira/browse/KYLIN-5761 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta Upgrade the embedded tomcat and netty to fix vulnerabilities -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5760) The API of get job list supports match model exactly
[
https://issues.apache.org/jira/browse/KYLIN-5760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831129#comment-17831129
]
pengfei.zhan commented on KYLIN-5760:
-
h1. Design
Add a parameter _'exact' to control whether match job list exactly. By default
is false to keep previous behavior. If it's true, then match all jobs of the
given model name(case insensitive)._
返回任务列表: {{{}GET [http://host:port/kylin/api/jobs]{}}}{{{}{}}}
> The API of get job list supports match model exactly
>
>
> Key: KYLIN-5760
> URL: https://issues.apache.org/jira/browse/KYLIN-5760
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0-beta
>
>
> h1. Description
> At present the API of list jobs (/kylin/api/jobs) will return all job lists
> that fuzzy match the given key. Sometimes, for better performance, the key is
> applied to match jobs exactly, however, it's not supported yet.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KYLIN-5760) The API of get job list supports match model exactly
[ https://issues.apache.org/jira/browse/KYLIN-5760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5760: Description: h1. Description At present the API of list jobs (/kylin/api/jobs) will return all job lists that fuzzy match the given key. Sometimes, for better performance, the key is applied to match jobs exactly, however, it's not supported yet. was: h1. Description At present the API of list jobs (/kylin/api/jobs) will return all job lists that fuzzy match the given key. Sometimes, for better performance, the key is applied to match jobs exactly, however, it's not supported jet. > The API of get job list supports match model exactly > > > Key: KYLIN-5760 > URL: https://issues.apache.org/jira/browse/KYLIN-5760 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > > h1. Description > At present the API of list jobs (/kylin/api/jobs) will return all job lists > that fuzzy match the given key. Sometimes, for better performance, the key is > applied to match jobs exactly, however, it's not supported yet. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5748) Skip build flat table for index build job if all new created indexes can be built from existed parent index
[ https://issues.apache.org/jira/browse/KYLIN-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831024#comment-17831024 ] ASF GitHub Bot commented on KYLIN-5748: --- Mrhs121 commented on PR #2162: URL: https://github.com/apache/kylin/pull/2162#issuecomment-2021131685 Your proposal will be invalid in this use case: if you add a new bitmap type metric to the model, you can build the new index separately, and the task type is also INDEX_BUIDL. If you skip building the dictionary, an error will occur when generating encoding columns for the flat table later. > Skip build flat table for index build job if all new created indexes can be > built from existed parent index > --- > > Key: KYLIN-5748 > URL: https://issues.apache.org/jira/browse/KYLIN-5748 > Project: Kylin > Issue Type: Improvement >Affects Versions: 5.0-beta >Reporter: Zhenning Zhang >Assignee: Zhenning Zhang >Priority: Major > > For an index build job, if all the new added indexes can be built from their > existed parent index. No index was built from new created flat table. > For this case, there won't be any data inconsistency across indexes. And > there is no need to build a new flat table to serve the stage of index build > by layer. > So I think we can skip stage BUILD DICT and stage GENERATE FLAT TABLE for an > index build job in condition that all new added indexes can be built from > their existed parent index -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] KYLIN-5748 Skip build flat table for index build job if all new created indexes can be built from existed parent index [kylin]
Mrhs121 commented on PR #2162: URL: https://github.com/apache/kylin/pull/2162#issuecomment-2021131685 Your proposal will be invalid in this use case: if you add a new bitmap type metric to the model, you can build the new index separately, and the task type is also INDEX_BUIDL. If you skip building the dictionary, an error will occur when generating encoding columns for the flat table later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@kylin.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (KYLIN-5760) The API of get job list supports match model exactly
pengfei.zhan created KYLIN-5760: --- Summary: The API of get job list supports match model exactly Key: KYLIN-5760 URL: https://issues.apache.org/jira/browse/KYLIN-5760 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta h1. Description At present the API of list jobs (/kylin/api/jobs) will return all job lists that fuzzy match the given key. Sometimes, for better performance, the key is applied to match jobs exactly, however, it's not supported jet. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5759) Modify "ExecutableContext#addRunningJob" to avoid schedule job more than once
[
https://issues.apache.org/jira/browse/KYLIN-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830888#comment-17830888
]
pengfei.zhan commented on KYLIN-5759:
-
h1. Design
this method _{{ExecutableContext#addRunningJob}}_ will be separated from the
current thread (runningJobThreads.put(executable.getId(),
Thread.currentThread())) . The purpose of this is that {{_addRunningJob_}} is
just a record of which tasks have been scheduled, used to determine the
subsequent do not be repeated scheduling, this should not add the current
thread (scheduling thread), but in the task is actually executed when the
current thread add.
> Modify "ExecutableContext#addRunningJob" to avoid schedule job more than once
> -
>
> Key: KYLIN-5759
> URL: https://issues.apache.org/jira/browse/KYLIN-5759
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0-beta
>
>
> When a task is scheduled, the task is logged into runningJobs and the current
> thread is logged into runningJobThreads, which is expected to be the thread
> that executes the task, but in fact it is the scheduler's thread that is
> logged, which results in subsequent attempts to interrupt the scheduler
> FetcherRunner when killing the task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KYLIN-5759) Modify "ExecutableContext#addRunningJob" to avoid schedule job more than once
pengfei.zhan created KYLIN-5759: --- Summary: Modify "ExecutableContext#addRunningJob" to avoid schedule job more than once Key: KYLIN-5759 URL: https://issues.apache.org/jira/browse/KYLIN-5759 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta When a task is scheduled, the task is logged into runningJobs and the current thread is logged into runningJobThreads, which is expected to be the thread that executes the task, but in fact it is the scheduler's thread that is logged, which results in subsequent attempts to interrupt the scheduler FetcherRunner when killing the task. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] (KYLIN-5758) TDS template modify to english
[ https://issues.apache.org/jira/browse/KYLIN-5758 ] pengfei.zhan deleted comment on KYLIN-5758: - was (Author: JIRAUSER294653): h1. Expectation the exported tds file does not contains Chinese characters > TDS template modify to english > -- > > Key: KYLIN-5758 > URL: https://issues.apache.org/jira/browse/KYLIN-5758 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > Attachments: 1.png > > > *Steps to reproduce the issue:* > # Go to a model and click “Export TDS”. > # Open up the file in a text editor and look at the bottom. There are > Chinese characters. See attachment. > I confirmed this issue is present in 4.5.4 and .11, and likely exists in > other versions. The TDS file seems to work fine though. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5758) TDS template modify to english
[ https://issues.apache.org/jira/browse/KYLIN-5758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5758: Attachment: 1.png > TDS template modify to english > -- > > Key: KYLIN-5758 > URL: https://issues.apache.org/jira/browse/KYLIN-5758 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > Attachments: 1.png > > > *Steps to reproduce the issue:* > # Go to a model and click “Export TDS”. > # Open up the file in a text editor and look at the bottom. There are > Chinese characters. See attachment. > I confirmed this issue is present in 4.5.4 and .11, and likely exists in > other versions. The TDS file seems to work fine though. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5758) TDS template modify to english
[ https://issues.apache.org/jira/browse/KYLIN-5758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830883#comment-17830883 ] pengfei.zhan commented on KYLIN-5758: - h1. Expectation the exported tds file does not contains Chinese characters !1.png! > TDS template modify to english > -- > > Key: KYLIN-5758 > URL: https://issues.apache.org/jira/browse/KYLIN-5758 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > Attachments: 1.png > > > *Steps to reproduce the issue:* > # Go to a model and click “Export TDS”. > # Open up the file in a text editor and look at the bottom. There are > Chinese characters. See attachment. > I confirmed this issue is present in 4.5.4 and .11, and likely exists in > other versions. The TDS file seems to work fine though. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KYLIN-5758) TDS template modify to english
[ https://issues.apache.org/jira/browse/KYLIN-5758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830881#comment-17830881 ] pengfei.zhan edited comment on KYLIN-5758 at 3/26/24 11:12 AM: --- h1. Expectation the exported tds file does not contains Chinese characters was (Author: JIRAUSER294653): h1. Expectation the exported tds file does not contains Chinese characters !1.png! > TDS template modify to english > -- > > Key: KYLIN-5758 > URL: https://issues.apache.org/jira/browse/KYLIN-5758 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > > *Steps to reproduce the issue:* > # Go to a model and click “Export TDS”. > # Open up the file in a text editor and look at the bottom. There are > Chinese characters. See attachment. > I confirmed this issue is present in 4.5.4 and .11, and likely exists in > other versions. The TDS file seems to work fine though. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5758) TDS template modify to english
[ https://issues.apache.org/jira/browse/KYLIN-5758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830881#comment-17830881 ] pengfei.zhan commented on KYLIN-5758: - h1. Expectation the exported tds file does not contains Chinese characters !1.png! > TDS template modify to english > -- > > Key: KYLIN-5758 > URL: https://issues.apache.org/jira/browse/KYLIN-5758 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > > *Steps to reproduce the issue:* > # Go to a model and click “Export TDS”. > # Open up the file in a text editor and look at the bottom. There are > Chinese characters. See attachment. > I confirmed this issue is present in 4.5.4 and .11, and likely exists in > other versions. The TDS file seems to work fine though. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] (KYLIN-5757) Optimization of the API of listing models
[ https://issues.apache.org/jira/browse/KYLIN-5757 ] pengfei.zhan deleted comment on KYLIN-5757: - was (Author: JIRAUSER294653): h1. Expectation the exported tds file does not contains Chinese characters !1.png! > Optimization of the API of listing models > -- > > Key: KYLIN-5757 > URL: https://issues.apache.org/jira/browse/KYLIN-5757 > Project: Kylin > Issue Type: Improvement > Components: Others >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > Attachments: 1.png > > > The API of listing models very slow at sometime, improve it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5757) Optimization of the API of listing models
[ https://issues.apache.org/jira/browse/KYLIN-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830880#comment-17830880 ] pengfei.zhan commented on KYLIN-5757: - h1. Expectation the exported tds file does not contains Chinese characters !1.png! > Optimization of the API of listing models > -- > > Key: KYLIN-5757 > URL: https://issues.apache.org/jira/browse/KYLIN-5757 > Project: Kylin > Issue Type: Improvement > Components: Others >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > Attachments: 1.png > > > The API of listing models very slow at sometime, improve it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5757) Optimization of the API of listing models
[ https://issues.apache.org/jira/browse/KYLIN-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5757: Attachment: 1.png > Optimization of the API of listing models > -- > > Key: KYLIN-5757 > URL: https://issues.apache.org/jira/browse/KYLIN-5757 > Project: Kylin > Issue Type: Improvement > Components: Others >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > Attachments: 1.png > > > The API of listing models very slow at sometime, improve it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5758) TDS template modify to english
pengfei.zhan created KYLIN-5758: --- Summary: TDS template modify to english Key: KYLIN-5758 URL: https://issues.apache.org/jira/browse/KYLIN-5758 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta *Steps to reproduce the issue:* # Go to a model and click “Export TDS”. # Open up the file in a text editor and look at the bottom. There are Chinese characters. See attachment. I confirmed this issue is present in 4.5.4 and .11, and likely exists in other versions. The TDS file seems to work fine though. -- This message was sent by Atlassian Jira (v8.20.10#820010)
