[jira] [Commented] (KYLIN-5023) Support cluster deployMode for Spark App which master is standalone

Sat, 03 Jul 2021 04:40:06 -0700 


    [ 
https://issues.apache.org/jira/browse/KYLIN-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17374003#comment-17374003
 ] 

ASF GitHub Bot commented on KYLIN-5023:
---------------------------------------

hit-lacus edited a comment on pull request #1677:
URL: https://github.com/apache/kylin/pull/1677#issuecomment-873371589


   ## Design 
   
   [ root cause and new design Chinese 
](https://www.notion.so/xxyu/Dev-Diary-KYLIN-5023-Kylin-4-deployMode-Cluster-80c269b757a74f9892551701e252ceec)
   
   ### Configuration
   
   Enable standaone cluster mode by following configuration.
   
   ```
   kylin.engine.spark.standalone.master.httpUrl=http://cdh-master:10031
   kylin.engine.spark-conf.spark.submit.deployMode=cluster
   kylin.engine.spark-conf.spark.shuffle.service.enabled=false
   kylin.engine.spark-conf.spark.master=spark://cdh-master:10030
   ```
   
   
   ### Log Output
   
   ```
   2021-07-03 17:33:19,757 DEBUG [BadQueryDetector] 
service.BadQueryDetector:148 : Detect bad query.
   2021-07-03 17:33:21,327 INFO  [Scheduler 959451607 Job 
d5225f31-3a64-4541-95be-47e3f6916f3a-248] deploy.SparkApplicationClient:54 : 
d5225f31-3a64-4541-95be-47e3f6916f3a state is SUBMITTED .
   2021-07-03 17:33:22,703 INFO  [FetcherRunner 1179123013-26] 
threadpool.DefaultFetcherRunner:117 : Job Fetcher: 1 should running, 1 actual 
running, 0 stopped, 0 ready, 11 already succeed, 2 error, 7 discarded, 0 others
   2021-07-03 17:33:31,350 INFO  [Scheduler 959451607 Job 
d5225f31-3a64-4541-95be-47e3f6916f3a-248] deploy.SparkApplicationClient:54 : 
d5225f31-3a64-4541-95be-47e3f6916f3a state is SUBMITTED .
   2021-07-03 17:33:37,138 DEBUG [http-bio-7070-exec-7] 
badquery.BadQueryHistoryManager:65 : Loaded 0 Bad Query(s)
   2021-07-03 17:33:41,372 INFO  [Scheduler 959451607 Job 
d5225f31-3a64-4541-95be-47e3f6916f3a-248] deploy.SparkApplicationClient:54 : 
d5225f31-3a64-4541-95be-47e3f6916f3a state is SUBMITTED .
   2021-07-03 17:33:51,396 INFO  [Scheduler 959451607 Job 
d5225f31-3a64-4541-95be-47e3f6916f3a-248] deploy.SparkApplicationClient:54 : 
d5225f31-3a64-4541-95be-47e3f6916f3a state is SUBMITTED .
   2021-07-03 17:33:52,703 INFO  [FetcherRunner 1179123013-26] 
threadpool.DefaultFetcherRunner:117 : Job Fetcher: 1 should running, 1 actual 
running, 0 stopped, 0 ready, 11 already succeed, 2 error, 7 discarded, 0 others
   2021-07-03 17:34:01,448 INFO  [Scheduler 959451607 Job 
d5225f31-3a64-4541-95be-47e3f6916f3a-248] deploy.SparkApplicationClient:54 : 
d5225f31-3a64-4541-95be-47e3f6916f3a state is SUBMITTED .
   2021-07-03 17:34:11,470 INFO  [Scheduler 959451607 Job 
d5225f31-3a64-4541-95be-47e3f6916f3a-248] deploy.SparkApplicationClient:54 : 
d5225f31-3a64-4541-95be-47e3f6916f3a state is FINISHED .
   2021-07-03 17:34:11,471 INFO  [Scheduler 959451607 Job 
d5225f31-3a64-4541-95be-47e3f6916f3a-248] utils.MetaDumpUtil:118 : Ready to 
load KylinConfig from uri: 
xiaoxiangyu@hdfs,path=hdfs://cdh-master:8020/LacusDir/xiaoxiangyu/xiaoxiangyu/LACUS_PRJ/job_tmp/d5225f31-3a64-4541-95be-47e3f6916f3a-01/meta
   2021-07-03 17:34:11,512 INFO  [Scheduler 959451607 Job 
d5225f31-3a64-4541-95be-47e3f6916f3a-248] common.KylinConfigBase:263 : Kylin 
Config was updated with kylin.metadata.url.identifier : xiaoxiangyu
   2021-07-03 17:34:11,512 INFO  [Scheduler 959451607 Job 
d5225f31-3a64-4541-95be-47e3f6916f3a-248] common.KylinConfigBase:263 : Kylin 
Config was updated with kylin.log.spark-executor-properties-file : 
/root/xiaoxiang.yu/Kylin4/apache-kylin-4.0.0-SNAPSHOT-bin/conf/spark-executor-log4j.properties
   ```
   
   ### Kylin UI
   
   
![image](https://user-images.githubusercontent.com/14030549/124352963-2f02f600-dc36-11eb-814b-b0369bb65894.png)
   
   
![image](https://user-images.githubusercontent.com/14030549/124352972-3d511200-dc36-11eb-9a6e-7f0e3d9ff4a7.png)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


> Support cluster deployMode for Spark App which master is standalone
> -------------------------------------------------------------------
>
>                 Key: KYLIN-5023
>                 URL: https://issues.apache.org/jira/browse/KYLIN-5023
>             Project: Kylin
>          Issue Type: Task
>          Components: Spark Engine
>    Affects Versions: v4.0.0-beta
>            Reporter: Xiaoxiang Yu
>            Assignee: Xiaoxiang Yu
>            Priority: Major
>             Fix For: v4.0.0
>
>
> Support cluster deployMode for Spark App which master is standalone



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to