[jira] [Comment Edited] (KYLIN-3341) SQL queries take up too much memory
[ https://issues.apache.org/jira/browse/KYLIN-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443819#comment-16443819 ] RenZhiMin edited comment on KYLIN-3341 at 4/23/18 5:39 AM: --- The query has 12 segments. When 14:34:27 SQL execution, and initialize the first segment, in 14:34:28, 686 when initializing the second segment, in 14:34:30, 104 when initializing the third segment, in 14:34:31, 434 when initializing the fourth segment, and then in 14:35:03 began to have a large number of request data back hbase, when 14:36:01 out of memory [~Shaofengshi] was (Author: renzm): The query has 12 segments. When 14:34:27 SQL execution, and initialize the first segment, in 14:34:28, 686 when initializing the second segment, in 14:34:30, 104 when initializing the third segment, in 14:34:31, 434 when initializing the fourth segment, and then in 14:35:03 began to have a large number of request data back hbase, when 14:36:01 out of memory [~Shaofengshi] > SQL queries take up too much memory > --- > > Key: KYLIN-3341 > URL: https://issues.apache.org/jira/browse/KYLIN-3341 > Project: Kylin > Issue Type: Improvement > Components: Storage - HBase >Affects Versions: v2.0.0 >Reporter: RenZhiMin >Priority: Major > Attachments: log.txt > > > Some SQL will take up too much memory in the application, the service is not > stable, after analysis because the SQL query need to scan the relevant > segemnt corresponding hbase table, is through the thread asynchronous request > related to SQL cuboid corresponding region. After each segment makes a > request, wait for the data to process the hbase callbacks. > But sometimes SQL involves more segments, and before all the segments make > requests, there is a large amount of data returned, leading to memory alarm > and service instability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3341) SQL queries take up too much memory
[ https://issues.apache.org/jira/browse/KYLIN-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443819#comment-16443819 ] RenZhiMin edited comment on KYLIN-3341 at 4/19/18 10:12 AM: The query has 12 segments. When 14:34:27 SQL execution, and initialize the first segment, in 14:34:28, 686 when initializing the second segment, in 14:34:30, 104 when initializing the third segment, in 14:34:31, 434 when initializing the fourth segment, and then in 14:35:03 began to have a large number of request data back hbase, when 14:36:01 out of memory [~Shaofengshi] was (Author: renzm): The query has 12 segments. When 14:34:27 SQL execution, and initialize the first segment, in 14:34:28, 686 when initializing the second segment, in 14:34:30, 104 when initializing the third segment, in 14:34:31, 434 when initializing the fourth segment, and then in 14:35:03 began to have a large number of request data back hbase, when 14:36:01 out of memory > SQL queries take up too much memory > --- > > Key: KYLIN-3341 > URL: https://issues.apache.org/jira/browse/KYLIN-3341 > Project: Kylin > Issue Type: Improvement > Components: Storage - HBase >Affects Versions: v2.0.0 >Reporter: RenZhiMin >Priority: Major > Attachments: log.txt > > > Some SQL will take up too much memory in the application, the service is not > stable, after analysis because the SQL query need to scan the relevant > segemnt corresponding hbase table, is through the thread asynchronous request > related to SQL cuboid corresponding region. After each segment makes a > request, wait for the data to process the hbase callbacks. > But sometimes SQL involves more segments, and before all the segments make > requests, there is a large amount of data returned, leading to memory alarm > and service instability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3341) SQL queries take up too much memory
[ https://issues.apache.org/jira/browse/KYLIN-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443819#comment-16443819 ] RenZhiMin edited comment on KYLIN-3341 at 4/19/18 10:11 AM: The query has 12 segments. When 14:34:27 SQL execution, and initialize the first segment, in 14:34:28, 686 when initializing the second segment, in 14:34:30, 104 when initializing the third segment, in 14:34:31, 434 when initializing the fourth segment, and then in 14:35:03 began to have a large number of request data back hbase, when 14:36:01 out of memory was (Author: renzm): The query has 12 segments. When 14:34:27 SQL execution, and initialize the first segment, in 14:34:28, 686 when initializing the second segment, in 14:34:30, 104 when initializing the third segment, in 14:34:31, 434 when initializing the fourth segment, and then in 14:35:03 began to have a large number of hbase request of return data, alarm when 14:36:01 memory > SQL queries take up too much memory > --- > > Key: KYLIN-3341 > URL: https://issues.apache.org/jira/browse/KYLIN-3341 > Project: Kylin > Issue Type: Improvement > Components: Storage - HBase >Affects Versions: v2.0.0 >Reporter: RenZhiMin >Priority: Major > Attachments: log.txt > > > Some SQL will take up too much memory in the application, the service is not > stable, after analysis because the SQL query need to scan the relevant > segemnt corresponding hbase table, is through the thread asynchronous request > related to SQL cuboid corresponding region. After each segment makes a > request, wait for the data to process the hbase callbacks. > But sometimes SQL involves more segments, and before all the segments make > requests, there is a large amount of data returned, leading to memory alarm > and service instability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3341) SQL queries take up too much memory
[ https://issues.apache.org/jira/browse/KYLIN-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443819#comment-16443819 ] RenZhiMin commented on KYLIN-3341: -- The query has 12 segments. When 14:34:27 SQL execution, and initialize the first segment, in 14:34:28, 686 when initializing the second segment, in 14:34:30, 104 when initializing the third segment, in 14:34:31, 434 when initializing the fourth segment, and then in 14:35:03 began to have a large number of hbase request of return data, alarm when 14:36:01 memory > SQL queries take up too much memory > --- > > Key: KYLIN-3341 > URL: https://issues.apache.org/jira/browse/KYLIN-3341 > Project: Kylin > Issue Type: Improvement > Components: Storage - HBase >Affects Versions: v2.0.0 >Reporter: RenZhiMin >Priority: Major > Attachments: log.txt > > > Some SQL will take up too much memory in the application, the service is not > stable, after analysis because the SQL query need to scan the relevant > segemnt corresponding hbase table, is through the thread asynchronous request > related to SQL cuboid corresponding region. After each segment makes a > request, wait for the data to process the hbase callbacks. > But sometimes SQL involves more segments, and before all the segments make > requests, there is a large amount of data returned, leading to memory alarm > and service instability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3341) SQL queries take up too much memory
[ https://issues.apache.org/jira/browse/KYLIN-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin updated KYLIN-3341: - Attachment: log.txt > SQL queries take up too much memory > --- > > Key: KYLIN-3341 > URL: https://issues.apache.org/jira/browse/KYLIN-3341 > Project: Kylin > Issue Type: Improvement > Components: Storage - HBase >Affects Versions: v2.0.0 >Reporter: RenZhiMin >Priority: Major > Attachments: log.txt > > > Some SQL will take up too much memory in the application, the service is not > stable, after analysis because the SQL query need to scan the relevant > segemnt corresponding hbase table, is through the thread asynchronous request > related to SQL cuboid corresponding region. After each segment makes a > request, wait for the data to process the hbase callbacks. > But sometimes SQL involves more segments, and before all the segments make > requests, there is a large amount of data returned, leading to memory alarm > and service instability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3341) SQL queries take up too much memory
RenZhiMin created KYLIN-3341: Summary: SQL queries take up too much memory Key: KYLIN-3341 URL: https://issues.apache.org/jira/browse/KYLIN-3341 Project: Kylin Issue Type: Improvement Components: Storage - HBase Affects Versions: v2.0.0 Reporter: RenZhiMin Some SQL will take up too much memory in the application, the service is not stable, after analysis because the SQL query need to scan the relevant segemnt corresponding hbase table, is through the thread asynchronous request related to SQL cuboid corresponding region. After each segment makes a request, wait for the data to process the hbase callbacks. But sometimes SQL involves more segments, and before all the segments make requests, there is a large amount of data returned, leading to memory alarm and service instability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3312) The global dictionary does not specify the cube distinction.
[ https://issues.apache.org/jira/browse/KYLIN-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421892#comment-16421892 ] RenZhiMin commented on KYLIN-3312: -- [~yimingliu] Please check it. > The global dictionary does not specify the cube distinction. > > > Key: KYLIN-3312 > URL: https://issues.apache.org/jira/browse/KYLIN-3312 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.0.0 >Reporter: RenZhiMin >Priority: Major > > In the cube design, the COUNT_DISTINCT indicator USES the global dictionary. > When building a cube, a dictionary file is generated at HDFS, and the > corresponding directory is: > "/ resources/GlobalDict/dict/database. The tableName/column/", at the same > time will generate metadata information in hbase metadata, metadata in the > hbase rowkey information:"/dict/database. The tableName/column/". The value > information specifies the corresponding HDFS path. > The problem is that the corresponding HDFS data and hbase data are not > deleted when the cube is deleted. Another HDFS data path and hbase metadata > rowkey information did not specify the concrete cube, if a table column in > more than one cube is to use global dictionary, can lead to multiple share a > dictionary data cube -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3296) When merge cube，get java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421886#comment-16421886 ] RenZhiMin commented on KYLIN-3296: -- [~Shaofengshi] please check. > When merge cube，get java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Assignee: RenZhiMin >Priority: Major > Labels: patch > Attachments: JIRA.master.3296.patch > > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3296) When merge cube，get java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412223#comment-16412223 ] RenZhiMin commented on KYLIN-3296: -- I have resubmitted it, please check. > When merge cube，get java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Priority: Major > Labels: patch > Attachments: JIRA.master.3296.patch > > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3296) When merge cube，get java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin updated KYLIN-3296: - Attachment: JIRA.master.3296.patch > When merge cube，get java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Priority: Major > Labels: patch > Attachments: JIRA.master.3296.patch > > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3296) When merge cube，get java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin updated KYLIN-3296: - Attachment: (was: JIRA.master.3296.patch) > When merge cube，get java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Priority: Major > Labels: patch > Attachments: JIRA.master.3296.patch > > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3312) The global dictionary does not specify the cube distinction.
RenZhiMin created KYLIN-3312: Summary: The global dictionary does not specify the cube distinction. Key: KYLIN-3312 URL: https://issues.apache.org/jira/browse/KYLIN-3312 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: v2.0.0 Reporter: RenZhiMin In the cube design, the COUNT_DISTINCT indicator USES the global dictionary. When building a cube, a dictionary file is generated at HDFS, and the corresponding directory is: "/ resources/GlobalDict/dict/database. The tableName/column/", at the same time will generate metadata information in hbase metadata, metadata in the hbase rowkey information:"/dict/database. The tableName/column/". The value information specifies the corresponding HDFS path. The problem is that the corresponding HDFS data and hbase data are not deleted when the cube is deleted. Another HDFS data path and hbase metadata rowkey information did not specify the concrete cube, if a table column in more than one cube is to use global dictionary, can lead to multiple share a dictionary data cube -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3296) cube合并时，java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410833#comment-16410833 ] RenZhiMin commented on KYLIN-3296: -- Hi [~yimingliu] I have manually generate a patch by git . Please check it. > cube合并时，java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Priority: Major > Labels: patch > Attachments: JIRA.master.3296.patch > > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3296) cube合并时，java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin updated KYLIN-3296: - Attachment: JIRA.master.3296.patch > cube合并时，java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Priority: Major > Labels: patch > Attachments: JIRA.master.3296.patch > > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3296) cube合并时，java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin updated KYLIN-3296: - Attachment: (was: JIRA.master.3296.patch) > cube合并时，java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Priority: Major > Labels: patch > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3296) cube合并时，java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409176#comment-16409176 ] RenZhiMin commented on KYLIN-3296: -- I have already submitted it again. Please check it. > cube合并时，java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Priority: Major > Labels: patch > Attachments: JIRA.master.3296.patch > > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (KYLIN-3303) 全局字典没有增加cube区分
[
https://issues.apache.org/jira/browse/KYLIN-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
RenZhiMin updated KYLIN-3303:
-
Comment: was deleted
(was: [~Shaofengshi] [~yimingliu] Please check it.)
> 全局字典没有增加cube区分
> --
>
> Key: KYLIN-3303
> URL: https://issues.apache.org/jira/browse/KYLIN-3303
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.0.0
>Reporter: RenZhiMin
>Priority: Major
>
> cube设计中，某个COUNT_DISTINCT指标使用全局字典。当构建cube时会在hdfs生成字典文件，对应的目录是：{kylin.env.hdfs-working-dir}/resources/GlobalDict/dict/database.tableName/column/，同时会在hbase元数据中生成元数据信息，hbase中元数据rowkey信息：/dict/database.tableName/column/。value信息指定对应的hdfs路径。问题是：当删除cube时没有删除对应的hdfs数据和hbase数据。另外hdfs数据路径和hbase元数据rowkey信息都没有指定具体的cube，如果某一个表的列在多个cube中都使用到了全局字典，会导致多个cube共用一份字典数据。
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (KYLIN-3296) cube合并时，java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin updated KYLIN-3296: - Comment: was deleted (was: [~Shaofengshi] I have already submitted it again. Please check it.) > cube合并时，java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Priority: Major > Labels: patch > Attachments: JIRA.master.3296.patch > > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3303) 全局字典没有增加cube区分
[
https://issues.apache.org/jira/browse/KYLIN-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407537#comment-16407537
]
RenZhiMin commented on KYLIN-3303:
--
[~Shaofengshi] [~yimingliu] Please check it.
> 全局字典没有增加cube区分
> --
>
> Key: KYLIN-3303
> URL: https://issues.apache.org/jira/browse/KYLIN-3303
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.0.0
>Reporter: RenZhiMin
>Priority: Major
>
> cube设计中，某个COUNT_DISTINCT指标使用全局字典。当构建cube时会在hdfs生成字典文件，对应的目录是：{kylin.env.hdfs-working-dir}/resources/GlobalDict/dict/database.tableName/column/，同时会在hbase元数据中生成元数据信息，hbase中元数据rowkey信息：/dict/database.tableName/column/。value信息指定对应的hdfs路径。问题是：当删除cube时没有删除对应的hdfs数据和hbase数据。另外hdfs数据路径和hbase元数据rowkey信息都没有指定具体的cube，如果某一个表的列在多个cube中都使用到了全局字典，会导致多个cube共用一份字典数据。
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3303) 全局字典没有增加cube区分
RenZhiMin created KYLIN-3303:
Summary: 全局字典没有增加cube区分
Key: KYLIN-3303
URL: https://issues.apache.org/jira/browse/KYLIN-3303
Project: Kylin
Issue Type: Bug
Components: Job Engine
Affects Versions: v2.0.0
Reporter: RenZhiMin
cube设计中，某个COUNT_DISTINCT指标使用全局字典。当构建cube时会在hdfs生成字典文件，对应的目录是：{kylin.env.hdfs-working-dir}/resources/GlobalDict/dict/database.tableName/column/，同时会在hbase元数据中生成元数据信息，hbase中元数据rowkey信息：/dict/database.tableName/column/。value信息指定对应的hdfs路径。问题是：当删除cube时没有删除对应的hdfs数据和hbase数据。另外hdfs数据路径和hbase元数据rowkey信息都没有指定具体的cube，如果某一个表的列在多个cube中都使用到了全局字典，会导致多个cube共用一份字典数据。
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3296) cube合并时，java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405774#comment-16405774 ] RenZhiMin commented on KYLIN-3296: -- [~Shaofengshi] I have already submitted it again. Please check it. > cube合并时，java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Priority: Major > Labels: patch > Attachments: JIRA.master.3296.patch > > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3296) cube合并时，java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin updated KYLIN-3296: - Attachment: JIRA.master.3296.patch > cube合并时，java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Priority: Major > Labels: patch > Attachments: JIRA.master.3296.patch > > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3296) cube合并时，java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin updated KYLIN-3296: - Attachment: (was: RowKeySplitter.patch) > cube合并时，java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Priority: Major > Labels: patch > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3296) cube合并时，java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin updated KYLIN-3296: - Component/s: Tools, Build and Test > cube合并时，java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Others, Tools, Build and Test >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Priority: Major > Labels: patch > Attachments: RowKeySplitter.patch > > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3296) cube合并时，java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin updated KYLIN-3296: - Labels: patch (was: ) > cube合并时，java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Others >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Priority: Major > Labels: patch > Attachments: RowKeySplitter.patch > > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3296) cube合并时，java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin updated KYLIN-3296: - Attachment: RowKeySplitter.patch > cube合并时，java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Others >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Priority: Major > Attachments: RowKeySplitter.patch > > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3296) cube合并时，java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
RenZhiMin created KYLIN-3296: Summary: cube合并时，java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method) Key: KYLIN-3296 URL: https://issues.apache.org/jira/browse/KYLIN-3296 Project: Kylin Issue Type: Bug Components: Others Affects Versions: v2.3.0 Reporter: RenZhiMin Attachments: RowKeySplitter.patch cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)” 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3269) 把project的权限授权给不存在用户时，出现错误。
[ https://issues.apache.org/jira/browse/KYLIN-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381600#comment-16381600 ] RenZhiMin commented on KYLIN-3269: -- 好的 > 把project的权限授权给不存在用户时，出现错误。 > -- > > Key: KYLIN-3269 > URL: https://issues.apache.org/jira/browse/KYLIN-3269 > Project: Kylin > Issue Type: Bug > Components: REST Service, Web >Affects Versions: v2.0.0 >Reporter: RenZhiMin >Priority: Major > Attachments: aclService.patch > > > 把project的权限授权给不存在用户时，会提示用户不存在，并且project上之前的权限都丢失了！所以应该在删除前先判断新的数据是否正确 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3269) 把project的权限授权给不存在用户时，出现错误。
[ https://issues.apache.org/jira/browse/KYLIN-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin updated KYLIN-3269: - Component/s: REST Service > 把project的权限授权给不存在用户时，出现错误。 > -- > > Key: KYLIN-3269 > URL: https://issues.apache.org/jira/browse/KYLIN-3269 > Project: Kylin > Issue Type: Bug > Components: REST Service, Web >Affects Versions: v2.0.0 >Reporter: RenZhiMin >Priority: Major > Attachments: aclService.patch > > > 把project的权限授权给不存在用户时，会提示用户不存在，并且project上之前的权限都丢失了！所以应该在删除前先判断新的数据是否正确 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3269) 把project的权限授权给不存在用户时，出现错误。
[ https://issues.apache.org/jira/browse/KYLIN-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin updated KYLIN-3269: - Flags: Patch > 把project的权限授权给不存在用户时，出现错误。 > -- > > Key: KYLIN-3269 > URL: https://issues.apache.org/jira/browse/KYLIN-3269 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: RenZhiMin >Priority: Major > Attachments: aclService.patch > > > 把project的权限授权给不存在用户时，会提示用户不存在，并且project上之前的权限都丢失了！ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3269) 把project的权限授权给不存在用户时，出现错误。
[ https://issues.apache.org/jira/browse/KYLIN-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin updated KYLIN-3269: - Attachment: aclService.patch > 把project的权限授权给不存在用户时，出现错误。 > -- > > Key: KYLIN-3269 > URL: https://issues.apache.org/jira/browse/KYLIN-3269 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: RenZhiMin >Priority: Major > Attachments: aclService.patch > > > 把project的权限授权给不存在用户时，会提示用户不存在，并且project上之前的权限都丢失了！ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3269) 把project的权限授权给不存在用户时，出现错误。
RenZhiMin created KYLIN-3269: Summary: 把project的权限授权给不存在用户时，出现错误。 Key: KYLIN-3269 URL: https://issues.apache.org/jira/browse/KYLIN-3269 Project: Kylin Issue Type: Bug Components: Web Affects Versions: v2.0.0 Reporter: RenZhiMin 把project的权限授权给不存在用户时，会提示用户不存在，并且project上之前的权限都丢失了！ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3252) 当hbase使用的文件系统和MapReduce不一样时，使用清理存储工具时，无法清理hbase所在文件系统的数据文件
[ https://issues.apache.org/jira/browse/KYLIN-3252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin updated KYLIN-3252: - Fix Version/s: v2.3.0 > 当hbase使用的文件系统和MapReduce不一样时，使用清理存储工具时，无法清理hbase所在文件系统的数据文件 > -- > > Key: KYLIN-3252 > URL: https://issues.apache.org/jira/browse/KYLIN-3252 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: RenZhiMin >Priority: Minor > Labels: patch > Fix For: v2.3.0 > > > 大家好，我使用的kylin版本是2.0，Hbase和MapReduce使用的文件系统不是同一个集群，当使用kylin清理存储工具时无法清理Hbase所在的文件系统上的数据 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3252) 当hbase使用的文件系统和MapReduce不一样时，使用清理存储工具时，无法清理hbase所在文件系统的数据文件
[ https://issues.apache.org/jira/browse/KYLIN-3252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin resolved KYLIN-3252. -- Resolution: Duplicate > 当hbase使用的文件系统和MapReduce不一样时，使用清理存储工具时，无法清理hbase所在文件系统的数据文件 > -- > > Key: KYLIN-3252 > URL: https://issues.apache.org/jira/browse/KYLIN-3252 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: RenZhiMin >Priority: Minor > Labels: patch > > 大家好，我使用的kylin版本是2.0，Hbase和MapReduce使用的文件系统不是同一个集群，当使用kylin清理存储工具时无法清理Hbase所在的文件系统上的数据 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3252) 当hbase使用的文件系统和MapReduce不一样时，使用清理存储工具时，无法清理hbase所在文件系统的数据文件
[ https://issues.apache.org/jira/browse/KYLIN-3252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RenZhiMin updated KYLIN-3252: - Description: 大家好，我使用的kylin版本是2.0，Hbase和MapReduce使用的文件系统不是同一个集群，当使用kylin清理存储工具时无法清理Hbase所在的文件系统上的数据 > 当hbase使用的文件系统和MapReduce不一样时，使用清理存储工具时，无法清理hbase所在文件系统的数据文件 > -- > > Key: KYLIN-3252 > URL: https://issues.apache.org/jira/browse/KYLIN-3252 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: RenZhiMin >Priority: Minor > Labels: patch > > 大家好，我使用的kylin版本是2.0，Hbase和MapReduce使用的文件系统不是同一个集群，当使用kylin清理存储工具时无法清理Hbase所在的文件系统上的数据 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3252) 当hbase使用的文件系统和MapReduce不一样时，使用清理存储工具时，无法清理hbase所在文件系统的数据文件
RenZhiMin created KYLIN-3252: Summary: 当hbase使用的文件系统和MapReduce不一样时，使用清理存储工具时，无法清理hbase所在文件系统的数据文件 Key: KYLIN-3252 URL: https://issues.apache.org/jira/browse/KYLIN-3252 Project: Kylin Issue Type: Bug Components: Tools, Build and Test Affects Versions: v2.0.0 Reporter: RenZhiMin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
