[jira] [Resolved] (KYLIN-2047) Ineffective null check in MetadataManager
[
https://issues.apache.org/jira/browse/KYLIN-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-2047.
Resolution: Fixed
Fix Version/s: v1.6.0
> Ineffective null check in MetadataManager
> -
>
> Key: KYLIN-2047
> URL: https://issues.apache.org/jira/browse/KYLIN-2047
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Dong Li
>Priority: Minor
> Fix For: v1.6.0
>
>
> {code}
> public DataModelDesc dropModel(DataModelDesc desc) throws IOException {
> logger.info("Dropping model '" + desc.getName() + "'");
> ResourceStore store = getStore();
> if (desc != null)
> store.deleteResource(desc.getResourcePath());
> {code}
> null check for desc is done after dereference, which is ineffective.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (KYLIN-2044) Unclosed DataInputByteBuffer in BitmapCounter#peekLength
[
https://issues.apache.org/jira/browse/KYLIN-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-2044.
Resolution: Fixed
Fix Version/s: v1.6.0
> Unclosed DataInputByteBuffer in BitmapCounter#peekLength
>
>
> Key: KYLIN-2044
> URL: https://issues.apache.org/jira/browse/KYLIN-2044
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Dong Li
>Priority: Minor
> Fix For: v1.6.0
>
>
> {code}
> DataInputByteBuffer input = new DataInputByteBuffer();
> input.reset(new ByteBuffer[] { in });
> MutableRoaringBitmap bitmap = new MutableRoaringBitmap();
> try {
> bitmap.deserialize(input);
> } catch (IOException e) {
> throw new IllegalStateException(e);
> }
> {code}
> input should be closed upon return from the method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (KYLIN-2045) Wrong argument order in JobInstanceExtractor#executeExtract()
[
https://issues.apache.org/jira/browse/KYLIN-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-2045.
Resolution: Fixed
Fix Version/s: v1.6.0
> Wrong argument order in JobInstanceExtractor#executeExtract()
> -
>
> Key: KYLIN-2045
> URL: https://issues.apache.org/jira/browse/KYLIN-2045
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Dong Li
> Fix For: v1.6.0
>
>
> {code}
> List jobInstances = listJobInstances(cube, project,
> startTime, endTime);
> {code}
> However, the order of first two arguments is reversed:
> {code}
> private List listJobInstances(String project, String cube,
> long startTime, long endTime) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (KYLIN-2033) Refactor broadcast of metadata change
[ https://issues.apache.org/jira/browse/KYLIN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang resolved KYLIN-2033. --- Resolution: Fixed Fix Version/s: v1.6.0 > Refactor broadcast of metadata change > - > > Key: KYLIN-2033 > URL: https://issues.apache.org/jira/browse/KYLIN-2033 > Project: Kylin > Issue Type: Improvement >Reporter: liyang > Fix For: v1.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (KYLIN-2033) Refactor broadcast of metadata change
[ https://issues.apache.org/jira/browse/KYLIN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang reassigned KYLIN-2033: - Assignee: liyang > Refactor broadcast of metadata change > - > > Key: KYLIN-2033 > URL: https://issues.apache.org/jira/browse/KYLIN-2033 > Project: Kylin > Issue Type: Improvement >Reporter: liyang >Assignee: liyang > Fix For: v1.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-2033) Refactor broadcast of metadata change
[ https://issues.apache.org/jira/browse/KYLIN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang updated KYLIN-2033: -- Summary: Refactor broadcast of metadata change (was: Refactor broadcast of metadata) > Refactor broadcast of metadata change > - > > Key: KYLIN-2033 > URL: https://issues.apache.org/jira/browse/KYLIN-2033 > Project: Kylin > Issue Type: Improvement >Reporter: liyang > Fix For: v1.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-2033) Refactor broadcast of metadata
[ https://issues.apache.org/jira/browse/KYLIN-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang updated KYLIN-2033: -- Summary: Refactor broadcast of metadata (was: Refactor broadcast to allow extensible entities) > Refactor broadcast of metadata > -- > > Key: KYLIN-2033 > URL: https://issues.apache.org/jira/browse/KYLIN-2033 > Project: Kylin > Issue Type: Improvement >Reporter: liyang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-2048) After save model, the new model dialog shows again.
liyang created KYLIN-2048: - Summary: After save model, the new model dialog shows again. Key: KYLIN-2048 URL: https://issues.apache.org/jira/browse/KYLIN-2048 Project: Kylin Issue Type: Bug Components: Web Affects Versions: v1.5.4 Reporter: liyang Assignee: Zhong,Jason After save model, the new model dialog shows again, which is not expected. Instead, it should go back to model landing page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-2032) Cube build failed when partition column isn't in dimension list
[
https://issues.apache.org/jira/browse/KYLIN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-2032:
Fix Version/s: v1.5.4.1
> Cube build failed when partition column isn't in dimension list
> ---
>
> Key: KYLIN-2032
> URL: https://issues.apache.org/jira/browse/KYLIN-2032
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v1.5.4
>Reporter: Shaofeng SHI
> Fix For: v1.5.4.1
>
>
> Create a cube which use "LO_DATE" as the partition date column, while it
> wasn't in the dimension list; when build the cube, it got failed in the "Fact
> distinct" step, mappers reported such exception:
> {code}
> Error: java.lang.IllegalArgumentException: Column SSB.V_LINEORDER.LO_DATE
> wasn't found on flat table. at
> org.apache.kylin.cube.model.CubeJoinedFlatTableDesc.getColumnIndex(CubeJoinedFlatTableDesc.java:145)
> at
> org.apache.kylin.cube.model.CubeJoinedFlatTableEnrich.getColumnIndex(CubeJoinedFlatTableEnrich.java:122)
> at
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapperBase.setup(FactDistinctColumnsMapperBase.java:80)
> at
> org.apache.kylin.engine.mr.steps.FactDistinctHiveColumnsMapper.setup(FactDistinctHiveColumnsMapper.java:59)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142) at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> {code}
> If add it to dimension, then the build got success;
> In CubeManager.getAllDictColumnsOnFact(), it said "add partition column in
> all case"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KYLIN-2026) NPE occurs when build a cube without partition column.
[ https://issues.apache.org/jira/browse/KYLIN-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15518016#comment-15518016 ] Shaofeng SHI commented on KYLIN-2026: - v1.5.4.1 will fix this, please keep tunned. > NPE occurs when build a cube without partition column. > -- > > Key: KYLIN-2026 > URL: https://issues.apache.org/jira/browse/KYLIN-2026 > Project: Kylin > Issue Type: Bug > Components: General >Affects Versions: v1.5.4 >Reporter: nichunen >Assignee: nichunen > Labels: patch > Fix For: v1.5.4.1 > > > I created a model with no partition column specified, step "Extract Fact > Table Distinct Columns" failed with MR log > "org.apache.hadoop.mapred.YarnChild: Exception running child : > java.lang.NullPointerException at > org.apache.kylin.cube.model.CubeJoinedFlatTableDesc.getColumnIndex(CubeJoinedFlatTableDesc.java:145)". > I doubt the root cause is in > org.apache.kylin.cube.CubeManager#getAllDictColumnsOnFact, a null value is > added to the return list due to > cubeDesc.getModel().getPartitionDesc().getPartitionDateColumnRef() gets null. > Attached is the patch to fix, only add a null check. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-2026) NPE occurs when build a cube without partition column.
[ https://issues.apache.org/jira/browse/KYLIN-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2026: Fix Version/s: (was: v1.5.4) v1.5.4.1 > NPE occurs when build a cube without partition column. > -- > > Key: KYLIN-2026 > URL: https://issues.apache.org/jira/browse/KYLIN-2026 > Project: Kylin > Issue Type: Bug > Components: General >Affects Versions: v1.5.4 >Reporter: nichunen >Assignee: nichunen > Labels: patch > Fix For: v1.5.4.1 > > > I created a model with no partition column specified, step "Extract Fact > Table Distinct Columns" failed with MR log > "org.apache.hadoop.mapred.YarnChild: Exception running child : > java.lang.NullPointerException at > org.apache.kylin.cube.model.CubeJoinedFlatTableDesc.getColumnIndex(CubeJoinedFlatTableDesc.java:145)". > I doubt the root cause is in > org.apache.kylin.cube.CubeManager#getAllDictColumnsOnFact, a null value is > added to the return list due to > cubeDesc.getModel().getPartitionDesc().getPartitionDateColumnRef() gets null. > Attached is the patch to fix, only add a null check. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-2026) NPE occurs when build a cube without partition column.
[ https://issues.apache.org/jira/browse/KYLIN-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2026: Attachment: (was: KYLIN-2026.patch) > NPE occurs when build a cube without partition column. > -- > > Key: KYLIN-2026 > URL: https://issues.apache.org/jira/browse/KYLIN-2026 > Project: Kylin > Issue Type: Bug > Components: General >Affects Versions: v1.5.4 >Reporter: nichunen >Assignee: nichunen > Labels: patch > Fix For: v1.5.4 > > > I created a model with no partition column specified, step "Extract Fact > Table Distinct Columns" failed with MR log > "org.apache.hadoop.mapred.YarnChild: Exception running child : > java.lang.NullPointerException at > org.apache.kylin.cube.model.CubeJoinedFlatTableDesc.getColumnIndex(CubeJoinedFlatTableDesc.java:145)". > I doubt the root cause is in > org.apache.kylin.cube.CubeManager#getAllDictColumnsOnFact, a null value is > added to the return list due to > cubeDesc.getModel().getPartitionDesc().getPartitionDateColumnRef() gets null. > Attached is the patch to fix, only add a null check. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-2047) Ineffective null check in MetadataManager
Ted Yu created KYLIN-2047:
-
Summary: Ineffective null check in MetadataManager
Key: KYLIN-2047
URL: https://issues.apache.org/jira/browse/KYLIN-2047
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
public DataModelDesc dropModel(DataModelDesc desc) throws IOException {
logger.info("Dropping model '" + desc.getName() + "'");
ResourceStore store = getStore();
if (desc != null)
store.deleteResource(desc.getResourcePath());
{code}
null check for desc is done after dereference, which is ineffective.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KYLIN-2046) Potential injected SQL attack vulnerability in QueryService
Ted Yu created KYLIN-2046:
-
Summary: Potential injected SQL attack vulnerability in
QueryService
Key: KYLIN-2046
URL: https://issues.apache.org/jira/browse/KYLIN-2046
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
{code}
String correctedSql = QueryUtil.massageSql(sqlRequest);
if (!correctedSql.equals(sqlRequest.getSql())) {
...
return execute(correctedSql, sqlRequest);
{code}
massageSql() uses regex to detect malformed SQL.
However, there may be SQL injection which is not detected by massageSql().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (KYLIN-2043) Rollback httpclient to 4.2.5 to align with Hadoop 2.6/2.7
[ https://issues.apache.org/jira/browse/KYLIN-2043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang resolved KYLIN-2043. --- Resolution: Fixed Assignee: liyang Fix Version/s: v1.6.0 > Rollback httpclient to 4.2.5 to align with Hadoop 2.6/2.7 > - > > Key: KYLIN-2043 > URL: https://issues.apache.org/jira/browse/KYLIN-2043 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: liyang > Fix For: v1.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-2045) Wrong argument order in JobInstanceExtractor#executeExtract()
Ted Yu created KYLIN-2045:
-
Summary: Wrong argument order in
JobInstanceExtractor#executeExtract()
Key: KYLIN-2045
URL: https://issues.apache.org/jira/browse/KYLIN-2045
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
{code}
List jobInstances = listJobInstances(cube, project,
startTime, endTime);
{code}
However, the order of first two arguments is reversed:
{code}
private List listJobInstances(String project, String cube,
long startTime, long endTime) {
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KYLIN-2044) Unclosed DataInputByteBuffer in BitmapCounter#peekLength
Ted Yu created KYLIN-2044:
-
Summary: Unclosed DataInputByteBuffer in BitmapCounter#peekLength
Key: KYLIN-2044
URL: https://issues.apache.org/jira/browse/KYLIN-2044
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
DataInputByteBuffer input = new DataInputByteBuffer();
input.reset(new ByteBuffer[] { in });
MutableRoaringBitmap bitmap = new MutableRoaringBitmap();
try {
bitmap.deserialize(input);
} catch (IOException e) {
throw new IllegalStateException(e);
}
{code}
input should be closed upon return from the method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KYLIN-2043) Rollback httpclient to 4.2.5 to align with Hadoop 2.6/2.7
liyang created KYLIN-2043: - Summary: Rollback httpclient to 4.2.5 to align with Hadoop 2.6/2.7 Key: KYLIN-2043 URL: https://issues.apache.org/jira/browse/KYLIN-2043 Project: Kylin Issue Type: Bug Reporter: liyang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-2042) QueryController puts entry in Cache w/o checking QueryCacheEnabled
Tajdar Siddiqui created KYLIN-2042: -- Summary: QueryController puts entry in Cache w/o checking QueryCacheEnabled Key: KYLIN-2042 URL: https://issues.apache.org/jira/browse/KYLIN-2042 Project: Kylin Issue Type: Improvement Components: REST Service Reporter: Tajdar Siddiqui Assignee: Zhong,Jason org.apache.kylin.rest.controller.QueryController around line 168 private SQLResponse doQueryWithCache(SQLRequest sqlRequest) cacheManager.getCache(SUCCESS_QUERY_CACHE).put(new Element(sqlRequest, sqlResponse)); This should be conditional on KylinConfig.getInstanceFromEnv().isQueryCacheEnabled() as well ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-2041) Wrong parameter definition in Get Hive Tables REST API
[ https://issues.apache.org/jira/browse/KYLIN-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy(Yiming) Liu resolved KYLIN-2041. -- Resolution: Fixed > Wrong parameter definition in Get Hive Tables REST API > -- > > Key: KYLIN-2041 > URL: https://issues.apache.org/jira/browse/KYLIN-2041 > Project: Kylin > Issue Type: Bug > Components: REST Service >Reporter: Billy(Yiming) Liu >Assignee: Billy(Yiming) Liu >Priority: Minor > Attachments: KYLIN-2041.patch > > > According to document: > http://kylin.apache.org/docs15/howto/howto_use_restapi.html#get-hive-tables > the "project" is a required parameter. But the source code is "required = > false" which will lead Internal Server Error when wrong parameter. The fix is > simple. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-1999) Use some compression at UT/IT
[ https://issues.apache.org/jira/browse/KYLIN-1999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy(Yiming) Liu resolved KYLIN-1999. -- Resolution: Fixed > Use some compression at UT/IT > - > > Key: KYLIN-1999 > URL: https://issues.apache.org/jira/browse/KYLIN-1999 > Project: Kylin > Issue Type: Improvement >Reporter: hongbin ma >Assignee: Billy(Yiming) Liu >Priority: Minor > Attachments: 0001-KYLIN-1999-for-UT-IT-enable-compress.patch > > > KYLIN-1984 disabled compression in packaging configurations to maximum ease > of use. Still we need to make sure everything will work if we want > compression enabled, especially for snappy. Currently in IT, hbase are > compressed by gzip, and mapreduce seems not enabling any compression -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-1995) Upgrade MapReduce properties which are deprecated
[ https://issues.apache.org/jira/browse/KYLIN-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy(Yiming) Liu resolved KYLIN-1995. -- > Upgrade MapReduce properties which are deprecated > - > > Key: KYLIN-1995 > URL: https://issues.apache.org/jira/browse/KYLIN-1995 > Project: Kylin > Issue Type: Task > Components: Job Engine >Affects Versions: v1.5.2 >Reporter: Billy(Yiming) Liu >Assignee: Billy(Yiming) Liu >Priority: Minor > Attachments: > 0001-KYLIN-1995-Upgrade-deprecated-properties-for-Hadoop-.patch, > 0002-KYLIN-1995-Upgrade-deprecated-properties-in-testcase.patch > > > Currently, Kylin use Hadoop 2.6 API. There are many properties are deprecated > in Hadoop 2.6, but still used in kylin's kylin_hive_conf.xml and > kylin_job_conf.xml, such as mapred.compress.map.output, > mapred.map.output.compression.codec and more. The full list is > https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html > > In this JIRA task, the deprecated properties will be upgraded to hadoop 2.6 > effective name and value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-2000) update document for KYLIN-1984
[ https://issues.apache.org/jira/browse/KYLIN-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy(Yiming) Liu resolved KYLIN-2000. -- Resolution: Fixed > update document for KYLIN-1984 > -- > > Key: KYLIN-2000 > URL: https://issues.apache.org/jira/browse/KYLIN-2000 > Project: Kylin > Issue Type: Improvement >Reporter: hongbin ma >Assignee: Billy(Yiming) Liu >Priority: Minor > Attachments: KYLIN-2000-Doc-for-compression-settings.patch > > > http://kylin.apache.org/docs15/install/advance_settings.html > the doc needs updating -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-1565) Read the kv max size from HBase config
[ https://issues.apache.org/jira/browse/KYLIN-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy(Yiming) Liu resolved KYLIN-1565. -- Resolution: Fixed > Read the kv max size from HBase config > -- > > Key: KYLIN-1565 > URL: https://issues.apache.org/jira/browse/KYLIN-1565 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: Billy(Yiming) Liu > Labels: newbie > Attachments: KYLIN-1565.patch > > > Kylin should try read the kv max size from HBase config rather than ask user > to repeat again in kylin.properties. > Suggest drop "kylin.hbase.client.keyvalue.maxsize", and read similar config > from HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-2019) Add license checker into checkstyle rule
[ https://issues.apache.org/jira/browse/KYLIN-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy(Yiming) Liu resolved KYLIN-2019. -- Resolution: Fixed > Add license checker into checkstyle rule > > > Key: KYLIN-2019 > URL: https://issues.apache.org/jira/browse/KYLIN-2019 > Project: Kylin > Issue Type: Improvement >Affects Versions: v1.5.4 >Reporter: Billy(Yiming) Liu >Assignee: Billy(Yiming) Liu >Priority: Minor > Attachments: > 0001-KYLIN-2019-add-license-checker-in-checkstyle-plugin.patch, > 0002-KYLIN-2019-Remove-unused-checkstyle-config.patch > > > Currently, not all files begin with the same format Apache Licence, > especially from different developers among the community. > In this update, the standard license file would be added into source repo, > and checkstyle plugin will check it in validate phase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-1978) kylin.sh compatible issue on Ubuntu
[
https://issues.apache.org/jira/browse/KYLIN-1978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy(Yiming) Liu resolved KYLIN-1978.
--
Resolution: Fixed
Fix Version/s: (was: Backlog)
> kylin.sh compatible issue on Ubuntu
> ---
>
> Key: KYLIN-1978
> URL: https://issues.apache.org/jira/browse/KYLIN-1978
> Project: Kylin
> Issue Type: Bug
> Components: Environment
>Affects Versions: v1.5.3
>Reporter: Shaofeng SHI
>Assignee: Billy(Yiming) Liu
> Attachments: KYLIN-1978.patch
>
>
> Reported by Marcelo(marcelo.n...@quantium.com.au) in the mailing list:
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=14.04
> DISTRIB_CODENAME=trusty
> DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS"
> NAME="Ubuntu"
> VERSION="14.04.4 LTS, Trusty Tahr"
> ID=ubuntu
> ID_LIKE=debian
> PRETTY_NAME="Ubuntu 14.04.4 LTS"
> VERSION_ID="14.04"
> HOME_URL="http://www.ubuntu.com/";
> SUPPORT_URL="http://help.ubuntu.com/";
> BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/";
> mapr@qtausc-vpcsdev04:~/kylin/apache-kylin-1.5.3-HBase1.x-bin/bin$ ./kylin.sh
> start
> *KYLIN_HOME is set to /home/mapr/kylin/apache-kylin-1.5.3-HBase1.x-bin*
> cat: invalid option -- '1'
> Try 'cat --help' for more information.
> -mkdir: Not enough arguments: expected 1 but got 0
> Usage: hadoop fs [generic options] -mkdir [-p] ...
> failed to create , Please make sure the user has right to access
> That is what is happening when I try to start kylin.
> I traced the error and the first one come from get-properties.sh at these
> line
> for i in `cat ${KYLIN_HOME}/conf/kylin.properties | grep -w "^$1" | grep -v
> '^#' | awk -F= '{ n = index($0,"="); print substr($0,n+1)}' | cut -c 1-`
> and as you can see kylin home is set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (KYLIN-2041) Wrong parameter definition in Get Hive Tables REST API
[ https://issues.apache.org/jira/browse/KYLIN-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy(Yiming) Liu reassigned KYLIN-2041: Assignee: Billy(Yiming) Liu (was: Zhong,Jason) > Wrong parameter definition in Get Hive Tables REST API > -- > > Key: KYLIN-2041 > URL: https://issues.apache.org/jira/browse/KYLIN-2041 > Project: Kylin > Issue Type: Bug > Components: REST Service >Reporter: Billy(Yiming) Liu >Assignee: Billy(Yiming) Liu >Priority: Minor > Attachments: KYLIN-2041.patch > > > According to document: > http://kylin.apache.org/docs15/howto/howto_use_restapi.html#get-hive-tables > the "project" is a required parameter. But the source code is "required = > false" which will lead Internal Server Error when wrong parameter. The fix is > simple. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-2041) Wrong parameter definition in Get Hive Tables REST API
[ https://issues.apache.org/jira/browse/KYLIN-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy(Yiming) Liu updated KYLIN-2041: - Attachment: KYLIN-2041.patch Patch is ready. > Wrong parameter definition in Get Hive Tables REST API > -- > > Key: KYLIN-2041 > URL: https://issues.apache.org/jira/browse/KYLIN-2041 > Project: Kylin > Issue Type: Bug > Components: REST Service >Reporter: Billy(Yiming) Liu >Assignee: Billy(Yiming) Liu >Priority: Minor > Attachments: KYLIN-2041.patch > > > According to document: > http://kylin.apache.org/docs15/howto/howto_use_restapi.html#get-hive-tables > the "project" is a required parameter. But the source code is "required = > false" which will lead Internal Server Error when wrong parameter. The fix is > simple. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-2041) Wrong parameter definition in Get Hive Tables REST API
Billy(Yiming) Liu created KYLIN-2041: Summary: Wrong parameter definition in Get Hive Tables REST API Key: KYLIN-2041 URL: https://issues.apache.org/jira/browse/KYLIN-2041 Project: Kylin Issue Type: Bug Components: REST Service Reporter: Billy(Yiming) Liu Assignee: Zhong,Jason Priority: Minor According to document: http://kylin.apache.org/docs15/howto/howto_use_restapi.html#get-hive-tables the "project" is a required parameter. But the source code is "required = false" which will lead Internal Server Error when wrong parameter. The fix is simple. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-1833) union operation will cause error result
[ https://issues.apache.org/jira/browse/KYLIN-1833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang resolved KYLIN-1833. --- Resolution: Fixed Fix Version/s: v1.5.4 > union operation will cause error result > --- > > Key: KYLIN-1833 > URL: https://issues.apache.org/jira/browse/KYLIN-1833 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v1.3.0, v1.5.2 >Reporter: fengYu >Assignee: fengYu > Fix For: v1.5.4 > > Attachments: 0001-KYLIN-1833-union-query-get-error-result.patch > > > query like this will get error result : > select * from ( > select 'b', count(1) from kylin_sales where lstg_format_name >= 'Auction' > union all > select 'a', count(1) from kylin_sales where lstg_format_name >= 'Others' > union all > select 'c', count(1) from kylin_sales where lstg_format_name >= 'FP-GTC' > ) order by 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-2040) Release 1.5.4.1
Shaofeng SHI created KYLIN-2040: --- Summary: Release 1.5.4.1 Key: KYLIN-2040 URL: https://issues.apache.org/jira/browse/KYLIN-2040 Project: Kylin Issue Type: Task Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1828) java.lang.StringIndexOutOfBoundsException in org.apache.kylin.storage.hbase.util.StorageCleanupJob
[ https://issues.apache.org/jira/browse/KYLIN-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang updated KYLIN-1828: -- Fix Version/s: (was: v1.5.4) v1.6.0 > java.lang.StringIndexOutOfBoundsException in > org.apache.kylin.storage.hbase.util.StorageCleanupJob > -- > > Key: KYLIN-1828 > URL: https://issues.apache.org/jira/browse/KYLIN-1828 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2.1 >Reporter: Richard Calaba >Assignee: Wang Cheng > Fix For: v1.6.0 > > Attachments: 0001-KYLIN-1828-StorageCleanupJob.patch > > > While running storage cleanup job: > ./bin/kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete > true > I see Hive tables in form > kylin_intermediate__1970010100_20160701031500 > in the defaul schema. > While running the above storage cleaner (v.1.5.2.1 - all previously built > Cubes Disabled & Dropped) I am getting an error: > 2016-06-27 22:28:08,480 INFO [main StorageCleanupJob:262]: Remove > intermediate hive table with job id fc44da88-cffc-4710-8726-ff910cf83451 with > job status ERROR > usage: StorageCleanupJob > -deleteDelete the unused storage > Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String > index out of range: -2 > at java.lang.String.substring(String.java:1904) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:269) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.run(StorageCleanupJob.java:91) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:308) > 2016-06-27 22:28:08,486 INFO [Thread-0 > HConnectionManager$HConnectionImplementation:1907]: Closing zookeeper > sessionid=0x154c97461586119 > 2016-06-27 22:28:08,491 INFO [Thread-0 ZooKeeper:684]: Session: > 0x154c97461586119 closed > 2016-06-27 22:28:08,491 INFO [main-EventThread ClientCnxn:509]: EventThread > shut down -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (KYLIN-1859) Use segment "uuid" instead of "name" to seek a segment across the system
[ https://issues.apache.org/jira/browse/KYLIN-1859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1859. --- > Use segment "uuid" instead of "name" to seek a segment across the system > > > Key: KYLIN-1859 > URL: https://issues.apache.org/jira/browse/KYLIN-1859 > Project: Kylin > Issue Type: Bug >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Minor > Fix For: v1.5.4 > > > The "uuid" is unique, while "name" isn't (name is composed by startTime, > endTime). It's better to use "uuid" in the job chains. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (KYLIN-1117) cluster management
[ https://issues.apache.org/jira/browse/KYLIN-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1117. --- Resolution: Incomplete After some evaluation, we decide to keep kylin as a thin client of hadoop; the clustering function should delegate to the hadoop cluster, so this function will not be released. > cluster management > -- > > Key: KYLIN-1117 > URL: https://issues.apache.org/jira/browse/KYLIN-1117 > Project: Kylin > Issue Type: Task > Components: General >Reporter: ZhouQianhao >Assignee: Shaofeng SHI > Fix For: Backlog > > Original Estimate: 1,344h > Remaining Estimate: 1,344h > > Kylin currently manages the cluster itself, so far it works fine since there > are only 2 roles for the cluster, job engine and query engine. > However when new features such as streaming is coming, deployment will be > more complex. > So new cluster management should be introduced. All the tasks followed or > related can be created as a sub-task -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-1859) Use segment "uuid" instead of "name" to seek a segment across the system
[ https://issues.apache.org/jira/browse/KYLIN-1859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-1859. - Resolution: Fixed Fix Version/s: v1.5.4 > Use segment "uuid" instead of "name" to seek a segment across the system > > > Key: KYLIN-1859 > URL: https://issues.apache.org/jira/browse/KYLIN-1859 > Project: Kylin > Issue Type: Bug >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Minor > Fix For: v1.5.4 > > > The "uuid" is unique, while "name" isn't (name is composed by startTime, > endTime). It's better to use "uuid" in the job chains. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (KYLIN-1983) Release 1.5.4
[ https://issues.apache.org/jira/browse/KYLIN-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1983. --- > Release 1.5.4 > - > > Key: KYLIN-1983 > URL: https://issues.apache.org/jira/browse/KYLIN-1983 > Project: Kylin > Issue Type: Task > Components: General >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Attachments: text.html > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (KYLIN-1038) Job retry on error
[ https://issues.apache.org/jira/browse/KYLIN-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1038. --- > Job retry on error > -- > > Key: KYLIN-1038 > URL: https://issues.apache.org/jira/browse/KYLIN-1038 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.4.0 >Reporter: hongbin ma >Assignee: Shaofeng SHI > Fix For: v1.5.3 > > > streaming cube will incur a lot of cube mergings, and some of them might fail > due to cluster resource shortage. proper reties would help to reduce manual > maintainance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-1038) Job retry on error
[ https://issues.apache.org/jira/browse/KYLIN-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-1038. - Resolution: Fixed Fix Version/s: (was: Backlog) v1.5.3 > Job retry on error > -- > > Key: KYLIN-1038 > URL: https://issues.apache.org/jira/browse/KYLIN-1038 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.4.0 >Reporter: hongbin ma >Assignee: Shaofeng SHI > Fix For: v1.5.3 > > > streaming cube will incur a lot of cube mergings, and some of them might fail > due to cluster resource shortage. proper reties would help to reduce manual > maintainance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-1983) Release 1.5.4
[ https://issues.apache.org/jira/browse/KYLIN-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-1983. - Resolution: Done > Release 1.5.4 > - > > Key: KYLIN-1983 > URL: https://issues.apache.org/jira/browse/KYLIN-1983 > Project: Kylin > Issue Type: Task > Components: General >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Attachments: text.html > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (KYLIN-1980) TopN issue: result not consistent with hive
[ https://issues.apache.org/jira/browse/KYLIN-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1980. --- Resolution: Not A Problem > TopN issue: result not consistent with hive > --- > > Key: KYLIN-1980 > URL: https://issues.apache.org/jira/browse/KYLIN-1980 > Project: Kylin > Issue Type: Bug >Reporter: Zhong Yanghong >Assignee: Shaofeng SHI > Attachments: hive_result.png, kylin_result.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-2037) Unable to clean hive intermediate tables by executing StorageCleanupJob.
[
https://issues.apache.org/jira/browse/KYLIN-2037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyang resolved KYLIN-2037.
---
Resolution: Duplicate
> Unable to clean hive intermediate tables by executing StorageCleanupJob.
> -
>
> Key: KYLIN-2037
> URL: https://issues.apache.org/jira/browse/KYLIN-2037
> Project: Kylin
> Issue Type: Bug
> Components: Storage - HBase
>Affects Versions: all
>Reporter: weiyongzhuang
>Assignee: liyang
> Fix For: Future
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> I try to clean intermediate data in Hive, HDFS, HBase by executing the
> command:
> "${KYLIN_HOME}/bin/kylin.sh
> org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete true", then I
> got the following error.
> usage: StorageCleanupJob
> -deleteDelete the unused storage
> Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String
> index out of range: -3
> at java.lang.String.substring(String.java:1904)
> at
> org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:290)
> at
> org.apache.kylin.storage.hbase.util.StorageCleanupJob.run(StorageCleanupJob.java:97)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:329)
> 2016-09-22 13:54:25,417 INFO [Thread-0
> ConnectionManager$HConnectionImplementation:1830]: Closing zookeeper
> sessionid=0x3574cdb7ca609a7
> 2016-09-22 13:54:25,420 INFO [Thread-0 ZooKeeper:684]: Session:
> 0x3574cdb7ca609a7 closed
> 2016-09-22 13:54:25,420 INFO [main-EventThread ClientCnxn:512]: EventThread
> shut down
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (KYLIN-1828) java.lang.StringIndexOutOfBoundsException in org.apache.kylin.storage.hbase.util.StorageCleanupJob
[ https://issues.apache.org/jira/browse/KYLIN-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang resolved KYLIN-1828. --- Resolution: Fixed Fix Version/s: (was: v1.6.0) v1.5.4 > java.lang.StringIndexOutOfBoundsException in > org.apache.kylin.storage.hbase.util.StorageCleanupJob > -- > > Key: KYLIN-1828 > URL: https://issues.apache.org/jira/browse/KYLIN-1828 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2.1 >Reporter: Richard Calaba >Assignee: Wang Cheng > Fix For: v1.5.4 > > Attachments: 0001-KYLIN-1828-StorageCleanupJob.patch > > > While running storage cleanup job: > ./bin/kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete > true > I see Hive tables in form > kylin_intermediate__1970010100_20160701031500 > in the defaul schema. > While running the above storage cleaner (v.1.5.2.1 - all previously built > Cubes Disabled & Dropped) I am getting an error: > 2016-06-27 22:28:08,480 INFO [main StorageCleanupJob:262]: Remove > intermediate hive table with job id fc44da88-cffc-4710-8726-ff910cf83451 with > job status ERROR > usage: StorageCleanupJob > -deleteDelete the unused storage > Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String > index out of range: -2 > at java.lang.String.substring(String.java:1904) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:269) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.run(StorageCleanupJob.java:91) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:308) > 2016-06-27 22:28:08,486 INFO [Thread-0 > HConnectionManager$HConnectionImplementation:1907]: Closing zookeeper > sessionid=0x154c97461586119 > 2016-06-27 22:28:08,491 INFO [Thread-0 ZooKeeper:684]: Session: > 0x154c97461586119 closed > 2016-06-27 22:28:08,491 INFO [main-EventThread ClientCnxn:509]: EventThread > shut down -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1828) java.lang.StringIndexOutOfBoundsException in org.apache.kylin.storage.hbase.util.StorageCleanupJob
[ https://issues.apache.org/jira/browse/KYLIN-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515978#comment-15515978 ] liyang commented on KYLIN-1828: --- Thanks Cheng, this is merged to master. > java.lang.StringIndexOutOfBoundsException in > org.apache.kylin.storage.hbase.util.StorageCleanupJob > -- > > Key: KYLIN-1828 > URL: https://issues.apache.org/jira/browse/KYLIN-1828 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2.1 >Reporter: Richard Calaba >Assignee: Wang Cheng > Fix For: v1.5.4 > > Attachments: 0001-KYLIN-1828-StorageCleanupJob.patch > > > While running storage cleanup job: > ./bin/kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete > true > I see Hive tables in form > kylin_intermediate__1970010100_20160701031500 > in the defaul schema. > While running the above storage cleaner (v.1.5.2.1 - all previously built > Cubes Disabled & Dropped) I am getting an error: > 2016-06-27 22:28:08,480 INFO [main StorageCleanupJob:262]: Remove > intermediate hive table with job id fc44da88-cffc-4710-8726-ff910cf83451 with > job status ERROR > usage: StorageCleanupJob > -deleteDelete the unused storage > Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String > index out of range: -2 > at java.lang.String.substring(String.java:1904) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:269) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.run(StorageCleanupJob.java:91) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:308) > 2016-06-27 22:28:08,486 INFO [Thread-0 > HConnectionManager$HConnectionImplementation:1907]: Closing zookeeper > sessionid=0x154c97461586119 > 2016-06-27 22:28:08,491 INFO [Thread-0 ZooKeeper:684]: Session: > 0x154c97461586119 closed > 2016-06-27 22:28:08,491 INFO [main-EventThread ClientCnxn:509]: EventThread > shut down -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KYLIN-1828) java.lang.StringIndexOutOfBoundsException in org.apache.kylin.storage.hbase.util.StorageCleanupJob
[ https://issues.apache.org/jira/browse/KYLIN-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515797#comment-15515797 ] Wang Cheng edited comment on KYLIN-1828 at 9/23/16 9:36 AM: Support clean up all the intermediate hive tables with setting "force" parameter. Examples: 1. kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete false --all false to check all intermediate hive tables can be deleted. 2. kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete true --all ture to delete all intermediate hive tables. was (Author: wormholer): Support clean up all the intermediate hive tables with setting "all" parameter. Examples: 1. kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete false --all false to check all intermediate hive tables can be deleted. 2. kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete true --all ture to delete all intermediate hive tables. > java.lang.StringIndexOutOfBoundsException in > org.apache.kylin.storage.hbase.util.StorageCleanupJob > -- > > Key: KYLIN-1828 > URL: https://issues.apache.org/jira/browse/KYLIN-1828 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2.1 >Reporter: Richard Calaba >Assignee: Wang Cheng > Fix For: v1.6.0 > > Attachments: 0001-KYLIN-1828-StorageCleanupJob.patch > > > While running storage cleanup job: > ./bin/kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete > true > I see Hive tables in form > kylin_intermediate__1970010100_20160701031500 > in the defaul schema. > While running the above storage cleaner (v.1.5.2.1 - all previously built > Cubes Disabled & Dropped) I am getting an error: > 2016-06-27 22:28:08,480 INFO [main StorageCleanupJob:262]: Remove > intermediate hive table with job id fc44da88-cffc-4710-8726-ff910cf83451 with > job status ERROR > usage: StorageCleanupJob > -deleteDelete the unused storage > Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String > index out of range: -2 > at java.lang.String.substring(String.java:1904) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:269) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.run(StorageCleanupJob.java:91) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:308) > 2016-06-27 22:28:08,486 INFO [Thread-0 > HConnectionManager$HConnectionImplementation:1907]: Closing zookeeper > sessionid=0x154c97461586119 > 2016-06-27 22:28:08,491 INFO [Thread-0 ZooKeeper:684]: Session: > 0x154c97461586119 closed > 2016-06-27 22:28:08,491 INFO [main-EventThread ClientCnxn:509]: EventThread > shut down -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KYLIN-1828) java.lang.StringIndexOutOfBoundsException in org.apache.kylin.storage.hbase.util.StorageCleanupJob
[ https://issues.apache.org/jira/browse/KYLIN-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515797#comment-15515797 ] Wang Cheng edited comment on KYLIN-1828 at 9/23/16 9:37 AM: Support clean up all the intermediate hive tables with setting "force" parameter. Examples: 1. kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete false --force false to check all intermediate hive tables can be deleted. 2. kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete true --force ture to delete all intermediate hive tables. was (Author: wormholer): Support clean up all the intermediate hive tables with setting "force" parameter. Examples: 1. kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete false --all false to check all intermediate hive tables can be deleted. 2. kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete true --all ture to delete all intermediate hive tables. > java.lang.StringIndexOutOfBoundsException in > org.apache.kylin.storage.hbase.util.StorageCleanupJob > -- > > Key: KYLIN-1828 > URL: https://issues.apache.org/jira/browse/KYLIN-1828 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2.1 >Reporter: Richard Calaba >Assignee: Wang Cheng > Fix For: v1.6.0 > > Attachments: 0001-KYLIN-1828-StorageCleanupJob.patch > > > While running storage cleanup job: > ./bin/kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete > true > I see Hive tables in form > kylin_intermediate__1970010100_20160701031500 > in the defaul schema. > While running the above storage cleaner (v.1.5.2.1 - all previously built > Cubes Disabled & Dropped) I am getting an error: > 2016-06-27 22:28:08,480 INFO [main StorageCleanupJob:262]: Remove > intermediate hive table with job id fc44da88-cffc-4710-8726-ff910cf83451 with > job status ERROR > usage: StorageCleanupJob > -deleteDelete the unused storage > Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String > index out of range: -2 > at java.lang.String.substring(String.java:1904) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:269) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.run(StorageCleanupJob.java:91) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:308) > 2016-06-27 22:28:08,486 INFO [Thread-0 > HConnectionManager$HConnectionImplementation:1907]: Closing zookeeper > sessionid=0x154c97461586119 > 2016-06-27 22:28:08,491 INFO [Thread-0 ZooKeeper:684]: Session: > 0x154c97461586119 closed > 2016-06-27 22:28:08,491 INFO [main-EventThread ClientCnxn:509]: EventThread > shut down -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1828) java.lang.StringIndexOutOfBoundsException in org.apache.kylin.storage.hbase.util.StorageCleanupJob
[ https://issues.apache.org/jira/browse/KYLIN-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang Cheng updated KYLIN-1828: -- Attachment: (was: 0001-KYLIN-1828-fix-StorageCleanupJob.patch) > java.lang.StringIndexOutOfBoundsException in > org.apache.kylin.storage.hbase.util.StorageCleanupJob > -- > > Key: KYLIN-1828 > URL: https://issues.apache.org/jira/browse/KYLIN-1828 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2.1 >Reporter: Richard Calaba >Assignee: Wang Cheng > Fix For: v1.6.0 > > Attachments: 0001-KYLIN-1828-StorageCleanupJob.patch > > > While running storage cleanup job: > ./bin/kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete > true > I see Hive tables in form > kylin_intermediate__1970010100_20160701031500 > in the defaul schema. > While running the above storage cleaner (v.1.5.2.1 - all previously built > Cubes Disabled & Dropped) I am getting an error: > 2016-06-27 22:28:08,480 INFO [main StorageCleanupJob:262]: Remove > intermediate hive table with job id fc44da88-cffc-4710-8726-ff910cf83451 with > job status ERROR > usage: StorageCleanupJob > -deleteDelete the unused storage > Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String > index out of range: -2 > at java.lang.String.substring(String.java:1904) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:269) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.run(StorageCleanupJob.java:91) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:308) > 2016-06-27 22:28:08,486 INFO [Thread-0 > HConnectionManager$HConnectionImplementation:1907]: Closing zookeeper > sessionid=0x154c97461586119 > 2016-06-27 22:28:08,491 INFO [Thread-0 ZooKeeper:684]: Session: > 0x154c97461586119 closed > 2016-06-27 22:28:08,491 INFO [main-EventThread ClientCnxn:509]: EventThread > shut down -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1828) java.lang.StringIndexOutOfBoundsException in org.apache.kylin.storage.hbase.util.StorageCleanupJob
[ https://issues.apache.org/jira/browse/KYLIN-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang Cheng updated KYLIN-1828: -- Attachment: 0001-KYLIN-1828-StorageCleanupJob.patch > java.lang.StringIndexOutOfBoundsException in > org.apache.kylin.storage.hbase.util.StorageCleanupJob > -- > > Key: KYLIN-1828 > URL: https://issues.apache.org/jira/browse/KYLIN-1828 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2.1 >Reporter: Richard Calaba >Assignee: Wang Cheng > Fix For: v1.6.0 > > Attachments: 0001-KYLIN-1828-StorageCleanupJob.patch, > 0001-KYLIN-1828-fix-StorageCleanupJob.patch > > > While running storage cleanup job: > ./bin/kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete > true > I see Hive tables in form > kylin_intermediate__1970010100_20160701031500 > in the defaul schema. > While running the above storage cleaner (v.1.5.2.1 - all previously built > Cubes Disabled & Dropped) I am getting an error: > 2016-06-27 22:28:08,480 INFO [main StorageCleanupJob:262]: Remove > intermediate hive table with job id fc44da88-cffc-4710-8726-ff910cf83451 with > job status ERROR > usage: StorageCleanupJob > -deleteDelete the unused storage > Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String > index out of range: -2 > at java.lang.String.substring(String.java:1904) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:269) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.run(StorageCleanupJob.java:91) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:308) > 2016-06-27 22:28:08,486 INFO [Thread-0 > HConnectionManager$HConnectionImplementation:1907]: Closing zookeeper > sessionid=0x154c97461586119 > 2016-06-27 22:28:08,491 INFO [Thread-0 ZooKeeper:684]: Session: > 0x154c97461586119 closed > 2016-06-27 22:28:08,491 INFO [main-EventThread ClientCnxn:509]: EventThread > shut down -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1828) java.lang.StringIndexOutOfBoundsException in org.apache.kylin.storage.hbase.util.StorageCleanupJob
[ https://issues.apache.org/jira/browse/KYLIN-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang Cheng updated KYLIN-1828: -- Attachment: (was: 0001-KYLIN-1828-fix-StorageCleanupJob.patch) > java.lang.StringIndexOutOfBoundsException in > org.apache.kylin.storage.hbase.util.StorageCleanupJob > -- > > Key: KYLIN-1828 > URL: https://issues.apache.org/jira/browse/KYLIN-1828 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2.1 >Reporter: Richard Calaba >Assignee: Wang Cheng > Fix For: v1.6.0 > > Attachments: 0001-KYLIN-1828-fix-StorageCleanupJob.patch > > > While running storage cleanup job: > ./bin/kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete > true > I see Hive tables in form > kylin_intermediate__1970010100_20160701031500 > in the defaul schema. > While running the above storage cleaner (v.1.5.2.1 - all previously built > Cubes Disabled & Dropped) I am getting an error: > 2016-06-27 22:28:08,480 INFO [main StorageCleanupJob:262]: Remove > intermediate hive table with job id fc44da88-cffc-4710-8726-ff910cf83451 with > job status ERROR > usage: StorageCleanupJob > -deleteDelete the unused storage > Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String > index out of range: -2 > at java.lang.String.substring(String.java:1904) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:269) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.run(StorageCleanupJob.java:91) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:308) > 2016-06-27 22:28:08,486 INFO [Thread-0 > HConnectionManager$HConnectionImplementation:1907]: Closing zookeeper > sessionid=0x154c97461586119 > 2016-06-27 22:28:08,491 INFO [Thread-0 ZooKeeper:684]: Session: > 0x154c97461586119 closed > 2016-06-27 22:28:08,491 INFO [main-EventThread ClientCnxn:509]: EventThread > shut down -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1828) java.lang.StringIndexOutOfBoundsException in org.apache.kylin.storage.hbase.util.StorageCleanupJob
[ https://issues.apache.org/jira/browse/KYLIN-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang Cheng updated KYLIN-1828: -- Attachment: 0001-KYLIN-1828-fix-StorageCleanupJob.patch Support clean up all the intermediate hive tables with setting "all" parameter. Examples: 1. kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete false --all false to check all intermediate hive tables can be deleted. 2. kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete true --all ture to delete all intermediate hive tables. > java.lang.StringIndexOutOfBoundsException in > org.apache.kylin.storage.hbase.util.StorageCleanupJob > -- > > Key: KYLIN-1828 > URL: https://issues.apache.org/jira/browse/KYLIN-1828 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2.1 >Reporter: Richard Calaba >Assignee: Wang Cheng > Fix For: v1.6.0 > > Attachments: 0001-KYLIN-1828-fix-StorageCleanupJob.patch, > 0001-KYLIN-1828-fix-StorageCleanupJob.patch > > > While running storage cleanup job: > ./bin/kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete > true > I see Hive tables in form > kylin_intermediate__1970010100_20160701031500 > in the defaul schema. > While running the above storage cleaner (v.1.5.2.1 - all previously built > Cubes Disabled & Dropped) I am getting an error: > 2016-06-27 22:28:08,480 INFO [main StorageCleanupJob:262]: Remove > intermediate hive table with job id fc44da88-cffc-4710-8726-ff910cf83451 with > job status ERROR > usage: StorageCleanupJob > -deleteDelete the unused storage > Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String > index out of range: -2 > at java.lang.String.substring(String.java:1904) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:269) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.run(StorageCleanupJob.java:91) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:308) > 2016-06-27 22:28:08,486 INFO [Thread-0 > HConnectionManager$HConnectionImplementation:1907]: Closing zookeeper > sessionid=0x154c97461586119 > 2016-06-27 22:28:08,491 INFO [Thread-0 ZooKeeper:684]: Session: > 0x154c97461586119 closed > 2016-06-27 22:28:08,491 INFO [main-EventThread ClientCnxn:509]: EventThread > shut down -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-2026) NPE occurs when build a cube without partition column.
[ https://issues.apache.org/jira/browse/KYLIN-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515641#comment-15515641 ] renfangming commented on KYLIN-2026: After add your patch，occur new bug in FactDistinctColumnsReducer 98 record, please check it Bug Detail as below: Error: java.lang.IndexOutOfBoundsException: Index: 9, Size: 9 at java.util.ArrayList.rangeCheck(ArrayList.java:635) at java.util.ArrayList.get(ArrayList.java:411) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.setup(FactDistinctColumnsReducer.java:98) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.setup(FactDistinctColumnsReducer.java:98) > NPE occurs when build a cube without partition column. > -- > > Key: KYLIN-2026 > URL: https://issues.apache.org/jira/browse/KYLIN-2026 > Project: Kylin > Issue Type: Bug > Components: General >Affects Versions: v1.5.4 >Reporter: nichunen >Assignee: nichunen > Labels: patch > Fix For: v1.5.4 > > Attachments: KYLIN-2026.patch > > > I created a model with no partition column specified, step "Extract Fact > Table Distinct Columns" failed with MR log > "org.apache.hadoop.mapred.YarnChild: Exception running child : > java.lang.NullPointerException at > org.apache.kylin.cube.model.CubeJoinedFlatTableDesc.getColumnIndex(CubeJoinedFlatTableDesc.java:145)". > I doubt the root cause is in > org.apache.kylin.cube.CubeManager#getAllDictColumnsOnFact, a null value is > added to the return list due to > cubeDesc.getModel().getPartitionDesc().getPartitionDateColumnRef() gets null. > Attached is the patch to fix, only add a null check. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
