[ https://issues.apache.org/jira/browse/KYLIN-4612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiaoxiang Yu resolved KYLIN-4612. --------------------------------- Resolution: Fixed > Support job status write to kafka > --------------------------------- > > Key: KYLIN-4612 > URL: https://issues.apache.org/jira/browse/KYLIN-4612 > Project: Kylin > Issue Type: Improvement > Components: Job Engine > Reporter: chuxiao > Assignee: chuxiao > Priority: Minor > Fix For: v3.1.1 > > > because more than hundrad job running , so job status changed write to kafka > instread of query job list. > English: > 1. The job status supports sending emails, but does not support other > notification systems. Sending emails is notifying people, and also needs to > support notifying other systems. For al scheduling system, before is to > prepare hive table data. Then build the kylin job and wait. After the job is > completed, it may be to notify the reporting system that the report query for > the segment date can be released, or may directly query the cube data to send > the daily report, or notify other systems in need. > If there are hundreds or thousands of build tasks running at the same time, > waiting for build notifications consumes less cost than the client's 10s > polling. > Producers and consumers who use kafka to build notifications can decouple > kylin from third-party systems, and third-party system problems will not > affect kyiln. > Compared with the real-time system cube, the notification is built to notify > kylin users, not to kylin administrators. The user does not care about the > details of the system cube, only the job result. In addition, due to > different consumers, the topic used to build status notifications and system > cubes are different, and even a Kafka cluster will not use a set. Therefore, > the Kafka write logic of the real-time system cube cannot be shared. > Since the third-party system may have a requirement to build a progress bar, > the status change of each subtask is also sent to kafka. > 2. Configuration. kylin.propertis adds > kylin.engine.job-status.write.kafka=true to enable this function. > Configure kylin.engine.job-status.kafka.bootstrap.servers=xxx and specify the > connection service address. > kylin.engine.job-status.kafka.topic.name=xx, specify the topic name to send. > Others require dekafka configuration, can be configured through > kylin.engine.job-status.kafka.{name}={value}, Kafka Properties configuration > will increase all the configuration of {name}={value}. default > ``` > ACKS_CONFIG, "-1", > COMPRESSION_TYPE_CONFIG=lz4, > RETRIES_CONFIG=3, > LINGER_MS_CONFIG=500, > BATCH_SIZE_CONFIG, 10000, > ``` > For parameters supported by other producers, see > http://kafka.apache.org/documentation/#producerconfigs > 3. How to receive. Parse the status from json. > The structure of the message body is as follows: > {"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f","jobName":"BUILD > CUBE-sys_probe-20150701000000_20190701000000-GMT+08:00 2019-07-05 > 11:42:33","status": "DISCARDED","subTaskSize": > "11","subTasks":[{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-00","jobName":"Create > Intermediate Flat Hive Table","status > ":"FINISHED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-01","jobName":"Extract > Fact Table Distinct Columns","status":"DISCARDED"},{"jobId > ":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-02","jobName":"Build Dimension > Dictionary","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f > -03","jobName":"Save Cuboid > Statistics","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-04","jobName":"Create > HTable" > ,"status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-05","jobName":"Build > Cube with Spark","status":"DISCARDED"},{ > "jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-06","jobName":"Convert Cuboid > Data to HFile","status":"DISCARDED"},{"jobId":"63d52094- > ca46-c4fa-7e77-242a6cf74f0f-07","jobName":"Load HFile to HBase > Table","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-08" > ,"jobName":"Update Cube > Info","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-09","jobName":"Hive > Cleanup","status > ":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-10","jobName":"Garbage > Collection on HDFS","status":"DISCARDED"}]} > ----------------------------------------------------- > 中文: > 1. > 构建执行状态支持发邮件,不支持其他主动通知方式。发邮件是通知人,也需要支持通知其他系统。对于外围调度系统来说,前序是准备hive表数据。再构建kylin作业,等待构建完成。构建完成后,后续是通知报表系统可以放开相应日期的报表查询了,或者直接查询构建数据发送日报,或者通知其他有需要的系统。 > 如果有几百上千个构建任务再同时运行,等待构建通知比客户端10s定时轮训消耗低。 > 用kafka做构建通知的生产者消费者,可以将kylin和第三方系统解耦,第三方系统异常不会影响kyiln。 > 跟实时系统cube相比,构建通知是通知kylin用户的,不是通知给kylin管理员的。用户不关心系统cube的细节,只关心构建结果。另外由于消费者不同,构建状态通知和系统cube用的topic不一样,甚至kafka集群都不会用一套。所以不能共用实时系统cube的kafka写入逻辑。 > 由于第三方系统可能有构建进度条需求,所以每个子task的状态变更也发送给kafka了。 > 2. 配置。kylin.propertis 增加kylin.engine.job-status.write.kafka=true,启用该功能。 > 配置kylin.engine.job-status.kafka.bootstrap.servers=xxx,指定连接服务地址。 > kylin.engine.job-status.kafka.topic.name=xx,指定发送topic名。 > 其他需要dekafka配置,可以通过kylin.engine.job-status.kafka.{name}={value}配置,kafka的Properties配置会增加所有的{name}={value}的配置。默认 > ``` > ACKS_CONFIG, "-1", > COMPRESSION_TYPE_CONFIG=lz4, > RETRIES_CONFIG=3, > LINGER_MS_CONFIG=500, > BATCH_SIZE_CONFIG, 10000, > ``` > 其他生产者支持的参数见 http://kafka.apache.org/documentation/#producerconfigs > 3. 如何接收。解析消息体的状态字段。 > 消息体结构如下: > {"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f","jobName":"BUILD CUBE - > sys_probe - 20150701000000_20190701000000 - GMT+08:00 2019-07-05 > 11:42:33","status":"DISCARDED","subTaskSize": > "11","subTasks":[{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-00","jobName":"Create > Intermediate Flat Hive > Table","status":"FINISHED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-01","jobName":"Extract > Fact Table Distinct > Columns","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-02","jobName":"Build > Dimension > Dictionary","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-03","jobName":"Save > Cuboid > Statistics","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-04","jobName":"Create > > HTable","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-05","jobName":"Build > Cube with > Spark","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-06","jobName":"Convert > Cuboid Data to > HFile","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-07","jobName":"Load > HFile to HBase > Table","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-08","jobName":"Update > Cube > Info","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-09","jobName":"Hive > > Cleanup","status":"DISCARDED"},{"jobId":"63d52094-ca46-c4fa-7e77-242a6cf74f0f-10","jobName":"Garbage > Collection on HDFS","status":"DISCARDED"}]} -- This message was sent by Atlassian Jira (v8.3.4#803005)