[
https://issues.apache.org/jira/browse/KYLIN-4612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chuxiao updated KYLIN-4612:
---------------------------
Description:
because more than hundrad job running , so job status changed write to kafka
instread of query job list.
1.构建执行状态支持发邮件，不支持其他主动通知方式。发邮件是通知人，也需要支持通知其他系统。对于外围调度系统来说，前序是准备hive表数据。再构建kylin作业，等待构建完成。构建完成后，后续是通知报表系统可以放开相应日期的报表查询了，或者直接查询构建数据发送日报，或者通知其他有需要的系统。
如果有几百上千个构建任务再同时运行，等待构建通知比客户端10s定时轮训消耗低。
用kafka做构建通知的生产者消费者，可以将kylin和第三方系统解耦，第三方系统异常不会影响kyiln。
跟实时系统cube相比，构建通知是通知kylin用户的，不是通知给kylin管理员的。用户不关心系统cube的细节，只关心构建结果。另外由于消费者不同，构建状态通知和系统cube用的topic不一样，甚至kafka集群都不会用一套。所以不能共用实时系统cube的kafka写入逻辑。
由于第三方系统可能有构建进度条需求，所以每个子task的状态变更也发送给kafka了。
2.配置。kylin.propertis 增加kylin.engine.job-status.write.kafka=true，启用该功能。
配置kylin.engine.job-status.kafka.bootstrap.servers=xxx，指定连接服务地址。
kylin.engine.job-status.kafka.topic.name=xx，指定发送topic名。
其他需要dekafka配置，可以通过kylin.engine.job-status.kafka.{name}={value}配置，kafka的Properties配置会增加所有的{name}={value}的配置。默认
ACKS_CONFIG, "-1",
COMPRESSION_TYPE_CONFIG=lz4，
RETRIES_CONFIG=3，
LINGER_MS_CONFIG=500，
BATCH_SIZE_CONFIG, 10000,
其他参数见 http://kafka.apache.org/documentation/#producerconfigs
3.如何接收。解析消息体的状态字段。
消息体结构如下：
{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f","jobName":"BUILD CUBE -
sys_probe - 20150701000000_20190701000000 - GMT+08:00 2019-07-05
11:42:33","status":"DISCARDED","subTaskSize":
"11","subTasks":[{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-00","jobName":"Create
Intermediate Flat Hive
Table","status":"FINISHED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-01","jobName":"Extract
Fact Table Distinct
Columns","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-02","jobName":"Build
Dimension
Dictionary","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-03","jobName":"Save
Cuboid
Statistics","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-04","jobName":"Create
HTable","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-05","jobName":"Build
Cube with
Spark","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-06","jobName":"Convert
Cuboid Data to
HFile","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-07","jobName":"Load
HFile to HBase
Table","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-08","jobName":"Update
Cube
Info","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-09","jobName":"Hive
Cleanup","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-10","jobName":"Garbage
Collection on HDFS","status":"DISCARDED"}]}
was:because more than hundrad job running , so job status changed write to
kafka instread of query job list
> Support job status write to kafka
> ---------------------------------
>
> Key: KYLIN-4612
> URL: https://issues.apache.org/jira/browse/KYLIN-4612
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
> Reporter: chuxiao
> Assignee: chuxiao
> Priority: Minor
>
> because more than hundrad job running , so job status changed write to kafka
> instread of query job list.
> 1.构建执行状态支持发邮件，不支持其他主动通知方式。发邮件是通知人，也需要支持通知其他系统。对于外围调度系统来说，前序是准备hive表数据。再构建kylin作业，等待构建完成。构建完成后，后续是通知报表系统可以放开相应日期的报表查询了，或者直接查询构建数据发送日报，或者通知其他有需要的系统。
> 如果有几百上千个构建任务再同时运行，等待构建通知比客户端10s定时轮训消耗低。
> 用kafka做构建通知的生产者消费者，可以将kylin和第三方系统解耦，第三方系统异常不会影响kyiln。
> 跟实时系统cube相比，构建通知是通知kylin用户的，不是通知给kylin管理员的。用户不关心系统cube的细节，只关心构建结果。另外由于消费者不同，构建状态通知和系统cube用的topic不一样，甚至kafka集群都不会用一套。所以不能共用实时系统cube的kafka写入逻辑。
> 由于第三方系统可能有构建进度条需求，所以每个子task的状态变更也发送给kafka了。
> 2.配置。kylin.propertis 增加kylin.engine.job-status.write.kafka=true，启用该功能。
> 配置kylin.engine.job-status.kafka.bootstrap.servers=xxx，指定连接服务地址。
> kylin.engine.job-status.kafka.topic.name=xx，指定发送topic名。
> 其他需要dekafka配置，可以通过kylin.engine.job-status.kafka.{name}={value}配置，kafka的Properties配置会增加所有的{name}={value}的配置。默认
> ACKS_CONFIG, "-1",
> COMPRESSION_TYPE_CONFIG=lz4，
> RETRIES_CONFIG=3，
> LINGER_MS_CONFIG=500，
> BATCH_SIZE_CONFIG, 10000,
> 其他参数见 http://kafka.apache.org/documentation/#producerconfigs
> 3.如何接收。解析消息体的状态字段。
> 消息体结构如下：
> {"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f","jobName":"BUILD CUBE -
> sys_probe - 20150701000000_20190701000000 - GMT+08:00 2019-07-05
> 11:42:33","status":"DISCARDED","subTaskSize":
> "11","subTasks":[{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-00","jobName":"Create
> Intermediate Flat Hive
> Table","status":"FINISHED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-01","jobName":"Extract
> Fact Table Distinct
> Columns","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-02","jobName":"Build
> Dimension
> Dictionary","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-03","jobName":"Save
> Cuboid
> Statistics","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-04","jobName":"Create
>
> HTable","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-05","jobName":"Build
> Cube with
> Spark","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-06","jobName":"Convert
> Cuboid Data to
> HFile","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-07","jobName":"Load
> HFile to HBase
> Table","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-08","jobName":"Update
> Cube
> Info","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-09","jobName":"Hive
>
> Cleanup","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-10","jobName":"Garbage
> Collection on HDFS","status":"DISCARDED"}]}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
