[jira] [Issue Comment Deleted] (KYLIN-3686) Top_N metric code requires cube storage type to be ID_SHARDED_HBASE, but the Web UI defaults to ID_HBASE and provides no safeguards against storage type mis
[
https://issues.apache.org/jira/browse/KYLIN-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-3686:
Comment: was deleted
(was: Let me check it.)
> Top_N metric code requires cube storage type to be ID_SHARDED_HBASE, but the
> Web UI defaults to ID_HBASE and provides no safeguards against storage type
> mismatch
> -
>
> Key: KYLIN-3686
> URL: https://issues.apache.org/jira/browse/KYLIN-3686
> Project: Kylin
> Issue Type: Improvement
> Components: Measure - TopN, Metadata, Web
>Affects Versions: v2.5.0
> Environment: HDP 2.5.6, Kylin 2.5
>Reporter: Vsevolod Ostapenko
>Priority: Major
>
> When new cube is defined via Kylin 2.5 UI, the default cube storage type is
> set to 0 (ID_HBASE).
> Top_N metric support is currently hard coded to expect cube storage type 2
> (ID_SHARDED_HBASE), and it *_does not_* check if the cube storage type is the
> "sharded HBASE".
> UI provides no safeguards either to prevent a user from defining a cube with
> Top_N metric that would blow up on the cube building stage with a perplexing
> stack trace like the following:
> {quote}2018-10-22 16:15:50,388 ERROR [main]
> org.apache.kylin.engine.mr.KylinMapper:
> java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey(NDCuboidBuilder.java:71)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:112)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:47)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> {quote}
> Please, either:
> – modify Top_N code to support all cube storage types (not only
> ID_SHARDED_HBASE),
> or
> – modify Top_N code to perform explicit check for cube storage type and
> raise descriptive exception, when cube storage is not the one that is
> expected. Plus update the UI to prevent the user from creating cube
> definitions that are incompatible with the storage type compatible with Top_N
> measure
> PS: NDCCuboidBuilder,java contains the following line:
> {quote}int offset = RowConstants.ROWKEY_SHARDID_LEN +
> RowConstants.ROWKEY_CUBOIDID_LEN; // skip shard and cuboidId{quote}
> If cube storage type is not ID_SHARDED_HBASE, offset is calculated
> incorrectly, which leads to ArrayIndexOutOfBounds exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-3695) Error while creating hive table through Kylin build cube with mysql imported tables
[
https://issues.apache.org/jira/browse/KYLIN-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI reassigned KYLIN-3695:
---
Assignee: XiaoXiang Yu
> Error while creating hive table through Kylin build cube with mysql imported
> tables
> ---
>
> Key: KYLIN-3695
> URL: https://issues.apache.org/jira/browse/KYLIN-3695
> Project: Kylin
> Issue Type: Bug
> Components: Integration
>Affects Versions: v2.5.0
>Reporter: Gaurav Rawat
>Assignee: XiaoXiang Yu
>Priority: Minor
>
> HI I am trying to build a cube with Kylin, the data gets souced fine from
> sqoop but the next step for creating hive tables fails . Looking at the
> command being fired it looks weird as the create statement looks good to me .
> I think the issue is with DOUBLE types as when I remove the same the create
> statement works fine . Can someone please help .
> I am using the stack in AWS EMR, kylin 2.5 hive 2.3.0
> The errors logs with commands as as below, the table is a msql table which
> had columns with DOUBLE type
> Command
> {code:java}
> hive -e "USE default; DROP TABLE IF EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368;
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 (
> HOLDINGS_STOCK_INVESTOR_ID string ,STOCK_INVESTORS_CHANNEL string
> ,STOCK_STOCK_ID string ,STOCK_DOMICILE string ,STOCK_STOCK_NM string
> ,STOCK_APPROACH string ,STOCK_STOCK_TYP string ,INVESTOR_ID string
> ,INVESTOR_NM string ,INVESTOR_DOMICILE_CNTRY string ,CLIENT_NM string
> ,INVESTOR_HOLDINGS_GROSS_ASSETS_USD double(22)
> ,INVESTOR_HOLDINGS_NET_ASSETS_USD double(22) ) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY '|' STORED AS TEXTFILE LOCATION
> 's3://wfg1tst-models/kylin/kylin_metadata/kylin-4ae3b18b-831b-da66-eb8c-7318245c4448/kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368';
> ALTER TABLE
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 SET
> TBLPROPERTIES('auto.purge'='true'); " --hiveconf hive.merge.mapredfiles=false
> --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2
> --hiveconf hive.exec.compress.output=true --hiveconf
> hive.auto.convert.join.noconditionaltask=true --hiveconf
> mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false
> --hiveconf hive.auto.convert.join.noconditionaltask.size=1 --hiveconf
> hive.stats.autogather=true{code}
> Error is as below
> {code:java}
> OK Time taken: 1.315 seconds OK Time taken: 0.09 seconds
> MismatchedTokenException(334!=347) at
> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
> at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at
> org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:6179)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:3808)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2382)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1333) at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204) at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:77) at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:70) at
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468) at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1316) at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1456) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1236) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1226) at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336) at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:787) at
> org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) at
> org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:136) FAILED: ParseException
> line 15:42 mismatched input '(' expecting ) near 'double' in create table
> statement{code}
>
[jira] [Assigned] (KYLIN-3686) Top_N metric code requires cube storage type to be ID_SHARDED_HBASE, but the Web UI defaults to ID_HBASE and provides no safeguards against storage type mismatch
[
https://issues.apache.org/jira/browse/KYLIN-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu reassigned KYLIN-3686:
---
Assignee: (was: XiaoXiang Yu)
> Top_N metric code requires cube storage type to be ID_SHARDED_HBASE, but the
> Web UI defaults to ID_HBASE and provides no safeguards against storage type
> mismatch
> -
>
> Key: KYLIN-3686
> URL: https://issues.apache.org/jira/browse/KYLIN-3686
> Project: Kylin
> Issue Type: Improvement
> Components: Measure - TopN, Metadata, Web
>Affects Versions: v2.5.0
> Environment: HDP 2.5.6, Kylin 2.5
>Reporter: Vsevolod Ostapenko
>Priority: Major
>
> When new cube is defined via Kylin 2.5 UI, the default cube storage type is
> set to 0 (ID_HBASE).
> Top_N metric support is currently hard coded to expect cube storage type 2
> (ID_SHARDED_HBASE), and it *_does not_* check if the cube storage type is the
> "sharded HBASE".
> UI provides no safeguards either to prevent a user from defining a cube with
> Top_N metric that would blow up on the cube building stage with a perplexing
> stack trace like the following:
> {quote}2018-10-22 16:15:50,388 ERROR [main]
> org.apache.kylin.engine.mr.KylinMapper:
> java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey(NDCuboidBuilder.java:71)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:112)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:47)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> {quote}
> Please, either:
> – modify Top_N code to support all cube storage types (not only
> ID_SHARDED_HBASE),
> or
> – modify Top_N code to perform explicit check for cube storage type and
> raise descriptive exception, when cube storage is not the one that is
> expected. Plus update the UI to prevent the user from creating cube
> definitions that are incompatible with the storage type compatible with Top_N
> measure
> PS: NDCCuboidBuilder,java contains the following line:
> {quote}int offset = RowConstants.ROWKEY_SHARDID_LEN +
> RowConstants.ROWKEY_CUBOIDID_LEN; // skip shard and cuboidId{quote}
> If cube storage type is not ID_SHARDED_HBASE, offset is calculated
> incorrectly, which leads to ArrayIndexOutOfBounds exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-3697) check port availability when starts kylin instance
[ https://issues.apache.org/jira/browse/KYLIN-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-3697: --- Assignee: May Zhou Fix Version/s: v2.6.0 > check port availability when starts kylin instance > -- > > Key: KYLIN-3697 > URL: https://issues.apache.org/jira/browse/KYLIN-3697 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.1 >Reporter: May Zhou >Assignee: May Zhou >Priority: Minor > Fix For: v2.6.0 > > > As I ran the script and it turned out with no error message, and I ran the > kylin.sh start script, the console output indicated that a kylin instance had > successfully started, but when I visit the :/kylin, I found > another service was running rather than kylin. > > I can only find the error message in the kylin.out, which is not so > user-friendly. > > Caused by: java.net.BindException: Address already in use (Bind failed) > at java.net.PlainSocketImpl.socketBind(Native Method) > at > java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387) > at java.net.ServerSocket.bind(ServerSocket.java:375) > I am wondering why check-env does not check the availability of port, and the > error message could be threw out more user-friendly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3693) TopN, Count distinct incorrect in Spark engine
[
https://issues.apache.org/jira/browse/KYLIN-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI reassigned KYLIN-3693:
---
Assignee: Shaofeng SHI
> TopN, Count distinct incorrect in Spark engine
> --
>
> Key: KYLIN-3693
> URL: https://issues.apache.org/jira/browse/KYLIN-3693
> Project: Kylin
> Issue Type: Bug
> Components: Spark Engine
>Affects Versions: v2.5.0, v2.5.1
>Reporter: Shaofeng SHI
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.5.2
>
>
> Long Chao firstly reported this bug; And then we found the currect
> integration test didn't cover this count distinct, topn measures with Spark
> engine. In "ITKylinQueryTest", some queries are not tested in the "left" join
> case, which means they were not covered the spark case. If we manually change
> to built the "left join" cube with spark, IT will report wrong result:
>
>
> {code:java}
> [ERROR] Failures:
> [ERROR]
> ITCombinationTest>ITKylinQueryTest.testPreciselyDistinctCountQuery:284->KylinTestBase.execAndCompQuery:544->KylinTestBase.execAndCompQuery:647->KylinTestBase.assertTableEquals:691
> value (table=query result of query01, row=0, col=USER_COUNT)
> expected:<8[46]> but was:<8[52]>
> [ERROR]
> ITCombinationTest>ITKylinQueryTest.testTopNQuery:277->KylinTestBase.execAndCompQuery:544->KylinTestBase.execAndCompQuery:647->KylinTestBase.assertTableEquals:691
> value (table=query result of query82, row=0, col=CAL_DT)
> expected:<2012-01-[28]> but was:<2012-01-[05]>
> [INFO]
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-3698) check-env.sh should print more details about checking items
[ https://issues.apache.org/jira/browse/KYLIN-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-3698: --- Assignee: May Zhou Fix Version/s: v2.6.0 > check-env.sh should print more details about checking items > --- > > Key: KYLIN-3698 > URL: https://issues.apache.org/jira/browse/KYLIN-3698 > Project: Kylin > Issue Type: Improvement >Affects Versions: v2.5.1 >Reporter: May Zhou >Assignee: May Zhou >Priority: Minor > Fix For: v2.6.0 > > > In the current version, when users run _check-env.sh_, if there's no error > message, it means everything is OK. > From my perspective, adding more details about the checking items when > executing check-env.sh is better. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit commented on issue #355: detailed configurations could be override in the Cube-level
asfgit commented on issue #355: detailed configurations could be override in the Cube-level URL: https://github.com/apache/kylin/pull/355#issuecomment-440662329 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] DingqianZhou opened a new pull request #355: detailed configurations could be override in the Cube-level
DingqianZhou opened a new pull request #355: detailed configurations could be override in the Cube-level URL: https://github.com/apache/kylin/pull/355 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3666) Mege cube step 2: Update dictionary throws IllegalStateException
[
https://issues.apache.org/jira/browse/KYLIN-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694495#comment-16694495
]
ASF subversion and git services commented on KYLIN-3666:
Commit 98cb504470c85718abd1616e78d36851c21f2fd0 in kylin's branch
refs/heads/master from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=98cb504 ]
KYLIN-3666 HDFS metadata url not be recognized
> Mege cube step 2: Update dictionary throws IllegalStateException
>
>
> Key: KYLIN-3666
> URL: https://issues.apache.org/jira/browse/KYLIN-3666
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.5.0
> Environment: cdh5.11.0
>Reporter: Xi Chen
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.2
>
>
> MERGE CUBE job failed in Step 2, throwing such exception:
>
> {code:java}
> java.lang.IllegalStateException: Metadata uri :
> kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
> is not recognized
> at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:172)
> at
> org.apache.kylin.common.KylinConfig.createInstanceFromUri(KylinConfig.java:182)
> at
> org.apache.kylin.engine.mr.steps.UpdateDictionaryStep.doWork(UpdateDictionaryStep.java:70)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: Metadata uri :
> kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
> looks like a file but it's neither a file nor a directory
> at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:162)
> ... 9 more
> {code}
> The metadataUrl is on hdfs, but decideUriType() method does not recognize it.
> Thus throwing such exception.
> Both MR and Spark engine are affected.
> Would [~Wayne0101] please have a look? I see it's related to KYLIN-3471
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-3665) Partition time column may never be added
[ https://issues.apache.org/jira/browse/KYLIN-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long reassigned KYLIN-3665: Assignee: Chao Long > Partition time column may never be added > > > Key: KYLIN-3665 > URL: https://issues.apache.org/jira/browse/KYLIN-3665 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > > The partition time column will never be added to the WHERE clause if the > partition date column is type of int or bigint and the date format is > "MMdd", "MMddHH", "MMddHHmm" or "MMddHHmmss". > Let’s say I have a fact table with two date partition column: day as the > format of integer (MMdd) and minute as the format of string (HHmm). I > have models created based on it and set the Partition Date Column to be “day” > with the format of MMdd and the Partition Time Column to be “minute” with > the format of HHmm. When I build the cube built on top of that model, I set > the Start Date to be 2018-11-01 00:00:00 and the End Date to be 2018-11-01 > 01:00:00. The building process is supposed to retrieve the data from the fact > table in the range of “day = 20181101 and minute>=’’ and minute<’0100’”. > However, it retrieves the data in the range of “WHERE 1=1 AND (DAY >= > 20181101 AND DAY < 20181101), so no data are actually retrieved. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3666) Mege cube step 2: Update dictionary throws IllegalStateException
[
https://issues.apache.org/jira/browse/KYLIN-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694494#comment-16694494
]
ASF GitHub Bot commented on KYLIN-3666:
---
shaofengshi closed pull request #347: KYLIN-3666 HDFS metadata url not be
recognized
URL: https://github.com/apache/kylin/pull/347
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java
index 6a9158d609..0624f94dc7 100644
---
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java
+++
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java
@@ -490,7 +490,14 @@ public static KylinConfig loadKylinPropsAndMetadata()
throws IOException {
public static KylinConfig
loadKylinConfigFromHdfs(SerializableConfiguration conf, String uri) {
HadoopUtil.setCurrentConfiguration(conf.get());
-return loadKylinConfigFromHdfs(uri);
+KylinConfig config = loadKylinConfigFromHdfs(uri);
+
+// This is a bad example where the thread local KylinConfig cannot be
auto-closed due to
+// limitation of MR API. It works because MR task runs its own
process. Do not copy.
+@SuppressWarnings("unused")
+SetAndUnsetThreadLocalConfig shouldAutoClose =
KylinConfig.setAndUnsetThreadLocalConfig(config);
+
+return config;
}
public static KylinConfig loadKylinConfigFromHdfs(String uri) {
@@ -517,11 +524,7 @@ public static KylinConfig loadKylinConfigFromHdfs(String
uri) {
} catch (IOException e) {
throw new RuntimeException(e);
}
-
-// This is a bad example where the thread local KylinConfig cannot be
auto-closed due to
-// limitation of MR API. It works because MR task runs its own
process. Do not copy.
-@SuppressWarnings("unused")
-SetAndUnsetThreadLocalConfig shouldAutoClose =
KylinConfig.setAndUnsetThreadLocalConfig(config);
+
kylinConfigCache.put(uri, config);
return config;
}
diff --git
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/UpdateDictionaryStep.java
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/UpdateDictionaryStep.java
index bcdb29f52c..28c567a7d0 100644
---
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/UpdateDictionaryStep.java
+++
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/UpdateDictionaryStep.java
@@ -43,6 +43,7 @@
import org.apache.kylin.cube.CubeUpdate;
import org.apache.kylin.dict.DictionaryInfo;
import org.apache.kylin.dict.DictionaryManager;
+import org.apache.kylin.engine.mr.common.AbstractHadoopJob;
import org.apache.kylin.engine.mr.common.BatchConstants;
import org.apache.kylin.job.exception.ExecuteException;
import org.apache.kylin.job.execution.AbstractExecutable;
@@ -67,7 +68,7 @@ protected ExecuteResult doWork(ExecutableContext context)
throws ExecuteExceptio
final String metadataUrl =
this.getParams().get(BatchConstants.ARG_META_URL);
final KylinConfig kylinConfHbase = cube.getConfig();
-final KylinConfig kylinConfHdfs =
KylinConfig.createInstanceFromUri(metadataUrl);
+final KylinConfig kylinConfHdfs =
AbstractHadoopJob.loadKylinConfigFromHdfs(metadataUrl);
Collections.sort(mergingSegments);
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Mege cube step 2: Update dictionary throws IllegalStateException
>
>
> Key: KYLIN-3666
> URL: https://issues.apache.org/jira/browse/KYLIN-3666
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.5.0
> Environment: cdh5.11.0
>Reporter: Xi Chen
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.2
>
>
> MERGE CUBE job failed in Step 2, throwing such exception:
>
> {code:java}
> java.lang.IllegalStateException: Metadata uri :
> kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
> is not recognized
> at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:172)
> at
>
[GitHub] shaofengshi closed pull request #347: KYLIN-3666 HDFS metadata url not be recognized
shaofengshi closed pull request #347: KYLIN-3666 HDFS metadata url not be
recognized
URL: https://github.com/apache/kylin/pull/347
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java
index 6a9158d609..0624f94dc7 100644
---
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java
+++
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java
@@ -490,7 +490,14 @@ public static KylinConfig loadKylinPropsAndMetadata()
throws IOException {
public static KylinConfig
loadKylinConfigFromHdfs(SerializableConfiguration conf, String uri) {
HadoopUtil.setCurrentConfiguration(conf.get());
-return loadKylinConfigFromHdfs(uri);
+KylinConfig config = loadKylinConfigFromHdfs(uri);
+
+// This is a bad example where the thread local KylinConfig cannot be
auto-closed due to
+// limitation of MR API. It works because MR task runs its own
process. Do not copy.
+@SuppressWarnings("unused")
+SetAndUnsetThreadLocalConfig shouldAutoClose =
KylinConfig.setAndUnsetThreadLocalConfig(config);
+
+return config;
}
public static KylinConfig loadKylinConfigFromHdfs(String uri) {
@@ -517,11 +524,7 @@ public static KylinConfig loadKylinConfigFromHdfs(String
uri) {
} catch (IOException e) {
throw new RuntimeException(e);
}
-
-// This is a bad example where the thread local KylinConfig cannot be
auto-closed due to
-// limitation of MR API. It works because MR task runs its own
process. Do not copy.
-@SuppressWarnings("unused")
-SetAndUnsetThreadLocalConfig shouldAutoClose =
KylinConfig.setAndUnsetThreadLocalConfig(config);
+
kylinConfigCache.put(uri, config);
return config;
}
diff --git
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/UpdateDictionaryStep.java
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/UpdateDictionaryStep.java
index bcdb29f52c..28c567a7d0 100644
---
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/UpdateDictionaryStep.java
+++
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/UpdateDictionaryStep.java
@@ -43,6 +43,7 @@
import org.apache.kylin.cube.CubeUpdate;
import org.apache.kylin.dict.DictionaryInfo;
import org.apache.kylin.dict.DictionaryManager;
+import org.apache.kylin.engine.mr.common.AbstractHadoopJob;
import org.apache.kylin.engine.mr.common.BatchConstants;
import org.apache.kylin.job.exception.ExecuteException;
import org.apache.kylin.job.execution.AbstractExecutable;
@@ -67,7 +68,7 @@ protected ExecuteResult doWork(ExecutableContext context)
throws ExecuteExceptio
final String metadataUrl =
this.getParams().get(BatchConstants.ARG_META_URL);
final KylinConfig kylinConfHbase = cube.getConfig();
-final KylinConfig kylinConfHdfs =
KylinConfig.createInstanceFromUri(metadataUrl);
+final KylinConfig kylinConfHdfs =
AbstractHadoopJob.loadKylinConfigFromHdfs(metadataUrl);
Collections.sort(mergingSegments);
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
